You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2018/12/22 00:10:24 UTC

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

vihang@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12118


Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................

IMPALA-7970 : Add support for automatic invalidates by polling metastore events

This change adds support to CatalogD to poll metastore events to issue invalidate
on tables automatically. It adds basic infrastructure to poll HMS notifications events
at a configurable frequency using a backend config called hms_event_polling_frequency_s
flag. Currently, it issues invalidate at tables when it received alter events on table
and partitions. It also adds tables/databases and removes tables from catalogD when it
receives create_table/create_database and drop_table/drop_database events. The default
value of hms_event_polling_frequency_s is 0 which disables the feature. A non-zero
value in seconds of this configuration can be used to enable the feature and set
the polling frequency.

The change also adds a new test class to test the basic functionality for each of the
event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
9 files changed, 1,014 insertions(+), 14 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/6
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@364
PS20, Line 364:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 00:13:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 8:

(3 comments)

I took a look at the locking aspect of this (given the history of the challenges there, we'll want multiple pairs of eyes on that aspect)

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG@18
PS8, Line 18: 
Can you summarise how this integrates with the existing catalog locking protocol? That's not mentioned in the design doc.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java@112
PS8, Line 112:     synchronized (thriftDb_) { thriftDb_.setMetastore_db(msDb); }
This additional ad-hoc locking without explanation of how it integrates with the other concurrency control in the catalog  (locks, version numbers, etc) is concerning.

If threads are updating this concurrently without using the protocol around versionLock_, that's probably going to mess up the whole protocol, since you'll see different thriftDb_ values for the same version number.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@305
PS8, Line 305:       catalog_.removeTable(oldTable.getDbName(), oldTable.getTableName());
Bharath mentioned it, but there's definitely a race here - you can see that removeTable() drops the lock before returning so this operation isn't atomic - there's a window where it would appear to another thread that the table disappeared.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Dec 2018 17:34:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
10 files changed, 1,350 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/11
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 6:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@305
PS6, Line 305:             "Unable to fetch the current notification event id from metastore. Metastore event processing will be disabled.");
> line too long (126 > 90)
fixed


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@321
PS6, Line 321:               "Unsupported alter_database event received. Event id %d, Database before : %s, Database after : %s",
> line too long (114 > 90)
fixed.


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@199
PS6, Line 199:         "Catalog should have a incomplete instance of table after CREATE_TABLE event is received",
> line too long (98 > 90)
fixed


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@208
PS6, Line 208:         "Catalog should have create a incomplete table after receiving CREATE_TABLE event",
> line too long (91 > 90)
fixed



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 00:17:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationFetchException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
14 files changed, 2,458 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/25
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 25
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 22:

(5 comments)

I forgot to publish these comments earlier. Publishing it here again just for the record.

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@383
PS20, Line 383: 
> Nit:
Done


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@360
PS22, Line 360:           throw new MetastoreNotificationException(debugString(
> To follow up on previous comments about errors. We may have the user busily
Thats a fair point. Let me add more details on what can go wrong during this event processing.
I would analyze what could be state of catalog at the event processing time based on the diagram I have in the event processor class.

Case I:
Catalog state is stale with respect to this event.
  a. It knows of a table with oldTblName but doesn't know about newTblName

This maps to the regular renameTable by remove oldTbl and add newTbl.

Case II:
Catalog state is exactly at the same state as of event.
 a. This means that catalog doesn't know of oldTbl. It knows about newTbl.

In this case also renameTable fails since it cannot find oldTbl. However, we can continue event processing. Basically, no action would be taken by this event and it needs to be ignored.

Case III:
Catalog state is ahead of event. This could mean a number of things.
 a. simple case is nothing exotic happens. OldTbl doesn't exist, newTbl exists --> case II

 b. user did a reverse rename from newtbl -> oldTbl post event. But we don't have any mechanism currently to distinguish this state from Case I without the version number support.

 c. User dropped the newTbl (or renamed it to something else). In this case, event processor will neither see a oldTbl or newTbl in the cache. It looks like there would be slight window of time where newTbl will appear until the actual event of the newTbl is processed and it is dropped (or renamed).

Given these various possibilities, I think the right thing to do is
Atomic rename by drop-and-add
  1. remove oldTbl if it exists.
  2. add newTbl if it doesn't exist. To compare a existing tbl we should use the creationTime and version.


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@515
PS22, Line 515:         LOG.info(debugString("Successfully removed database %s", dbName_));
> Briefly looked at these other error handling blocks. The exception/logging 
Thanks for taking a relook


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@334
PS21, Line 334:           lastSyncedEventId_ = event.getEventId();
              :         }
              :       }
              :     } catch (MetastoreNotificationException | CatalogException ex) {
              :       updateStatus(EventProcessorStatus.E
> This blocks invalidate metadata since both of them try to get a writelock a
changed the locking logic to use simple synchronized blocks. The code becomes a little more ugly but reduces the total blocking time for reset() as low as possible.


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@258
PS22, Line 258:   public synchronized void start(long fromEventId) {
> This is synchronized, which suggests that we might get multiple concurrent 
Both the start() and stop() methods do a Preconditions.checkState(). In this particular start method the check is on line 261. I will move it to the first couple of lines to improve readability.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jan 2019 23:24:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1677/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 01:26:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1696/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jan 2019 18:29:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 6:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@305
PS6, Line 305:             "Unable to fetch the current notification event id from metastore. Metastore event processing will be disabled.");
line too long (126 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@321
PS6, Line 321:               "Unsupported alter_database event received. Event id %d, Database before : %s, Database after : %s",
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@199
PS6, Line 199:         "Catalog should have a incomplete instance of table after CREATE_TABLE event is received",
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/6/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@208
PS6, Line 208:         "Catalog should have create a incomplete table after receiving CREATE_TABLE event",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 00:11:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,959 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/21
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. Be default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
10 files changed, 1,585 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/14
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#27). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationFetchException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
14 files changed, 2,551 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/27
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 27
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 11:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/1692/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 11
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Dec 2018 02:21:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 22:

(3 comments)

Thanks much for your great work in getting this in and helping us anticipate all the ways that things might go wrong in a production system. Thanks much also for discussing the code review comments in person.

This review has just a few minor follow-on comments. After that, I suspect we'll be in good shape to merge the code: we are reaching the point of diminishing returns in reviews; the next step is to try the feature out in a live cluster, and for that you'll need the code merged. A feature flag lets us turn off the feature if we encounter anything missed during reviews.

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@360
PS22, Line 360:           throw new MetastoreNotificationException(debugString(
To follow up on previous comments about errors. We may have the user busily hammering the system with INVALIDATEs and queries, each of which may mutate the cache ahead of the event processing here. If we encounter such a case, we should soldier on rather than stopping event processing.

So, the case that the old table does not exist, is that an expected error if the user issues an INVALIDATE? Or, does it mean that we have a likely bug and we should stop event processing?

The case of the missing new table is different: it would seem that there would be no good reason not to add the new table.

But, I wonder, does the renameTable() method handle the case in which the rename was already done? Something like:

* Rename table in HMS.
* Issue a query against the new name, causing the catalog to load metadata for the new table.
* Get the rename event.
* Find the old table and remove it, but be unable to add a new table since a table of that name was already loaded.


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@515
PS22, Line 515:         LOG.info(debugString("Successfully removed database %s", dbName_));
Briefly looked at these other error handling blocks. The exception/logging decisions look right on each of them.


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@258
PS22, Line 258:   public synchronized void start(long fromEventId) {
This is synchronized, which suggests that we might get multiple concurrent start/stop calls from different threads.

Should the start() (and stop()) methods check if they are in the proper state before proceeding? That is, should this method check if we are already in the ACTIVE state and simply return if so?



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jan 2019 21:18:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 25:

(10 comments)

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@308
PS25, Line 308:       LOG.info(String.format("Metastore event processing is disabled. Event polling "
> nit: fix formatting.
Done


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@145
PS25, Line 145: no
> not
Done


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@166
PS25, Line 166: drop event later
> Do we also need to remove the drop event or are we just letting it fail sil
drop events are implemented as dropIfExists. So since the create event is skipped, the drop event processing becomes a no-op. I wanted to remove the drop event as well, but it was complicating this code here without any added benefit. I will add a comment here saying why its ok to not remove the drop event so that its clear in the future.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@174
PS25, Line 174:         fromIndex++;
> Log how many events got filtered out? Helps diagnosing incase there are any
Good point. Also, added logging in the isRemovedAfter impl for createTable and createDatabase events.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@356
PS25, Line 356: if (this.dbName_.equalsIgnoreCase(dropTableEvent.dbName_) && this.tblName_
              :               .equalsIgnoreCase(dropTableEvent.tblName_)) {
              :             return true;
              :           }
> nit: simplify to return dbName_.equals() && tblName_.equals();
Actually, added a log statement in the if block so can't do this anymore.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@360
PS25, Line 360: else if (event.eventType_.equals(MetastoreEventType.ALTER_TABLE)) {
              :           // if this create table was renamed to some other name in the future return true
              :           AlterTableEvent alterTableEvent = (AlterTableEvent) event;
              :           if (alterTableEvent.isRename_ && this.dbName_
              :               .equalsIgnoreCase(alterTableEvent.tableBefore_.getDbName()) && this.tblName_
              :               .equalsIgnoreCase(alterTableEvent.tableBefore_.getTableName())) {
              :             return true;
              :           }
> clarify why rename qualifies as an inverse op for create?
Done


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@295
PS25, Line 295: LOG.info(String
              :           .format("Received %d events. Start event id : %d", response.getEvents().size(),
              :               lastSyncedEventId_));
> nit: format to fewer lines.
this statement cannot be converted to a 2 line statement without increasing the line width beyond 100


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@319
PS25, Line 319:   LOG.warn(String.format(
              :             "Not processing notification events since event processing status "
              :                 + "is %s. Last synced event id is %d", currentStatus,
              :             lastSyncedEventId_));
> nit: format to fewer lines.
Reduced the text verbosity to make it to 2 lines.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@339
PS25, Line 339: 
              :     // remove some partitions
              :     // change some partitions
> ?
had kept a reminder for myself to add partition and alter partition test. Added them. Thanks for pointing this out.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@583
PS25, Line 583:  in
> ?
updated the doc.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 25
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 01:18:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................

IMPALA-7970 : Add support for automatic invalidates by polling metastore events

This change adds support to CatalogD to poll metastore events to issue invalidate
on tables automatically. It adds basic infrastructure to poll HMS notifications events
at a configurable frequency using a backend config called hms_event_polling_frequency_s
flag. Currently, it issues invalidate at tables when it received alter events on table
and partitions. It also adds tables/databases and removes tables from catalogD when it
receives create_table/create_database and drop_table/drop_database events. The default
value of hms_event_polling_frequency_s is 0 which disables the feature. A non-zero
value in seconds of this configuration can be used to enable the feature and set
the polling frequency.

The change also adds a new test class to test the basic functionality for each of the
event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
9 files changed, 1,052 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/8
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java@33
PS20, Line 33:    * Get the current event id. Useful for restarting the event processing from a event id
> Maybe " on the HMS". Just to be clear that it's not the last event ID proce
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 01:11:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 9:

(39 comments)

This will be a great feature. Detailed review from the perspective of 1) synchronization and 2) diagnosing issues when they occur in the field.

This is a partial review, will review remaining files separately.

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@302
PS9, Line 302:     if (BackendConfig.INSTANCE.getHMSPollingFrequencyInSeconds() <= 0) {
Good check. Better to define this as behavior: to disable the polling, set the interval to 0 (or, trivially <0). This would be a good fall-back safety-valve for support cases, tests, etc.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@313
PS9, Line 313:       } catch (TException e) {
Nice use of try-with-catch. No need for nested tries. Remove inner catch, move catch to outer try.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2317
PS9, Line 2317:   MetastoreEventsProcessor getMetastoreEventProcessor() {
Should this be public? If only protected, please explicitly label it protected.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/Db.java@112
PS9, Line 112:     synchronized (thriftDb_) { thriftDb_.setMetastore_db(msDb); }
What is the synchronization model?

As written, we lock on the prior DB to set the new one. This means access is not synchronized. Better to add synchronized to the method itself, here and accessor.

But, more broadly, if we synchronize only at the DB level, we'll have all manner of race conditions. One client will grab the DB with one version the thrift object, then another will be slotted in underneath.

This probably needs more thought. Is the model documented in the write-up?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/Db.java@159
PS9, Line 159:     synchronized (thriftDb_) { return thriftDb_.deepCopy(); }
How often do we do this? Do we want this to block on an HMS call about parameters?

Maybe a better model is for the RPC calls to do this:

- Obtain a lock
- Get a pointer to the current thrift object
- Release the lock
- Make the RPC

Not that there is no need to copy the thrift object above: the logic here implies that the Thrift object is immutable once set. All we are doing is making sure we use the same object throughout the RPC.

Still, not convinced we have the right locking semantics.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/Db.java@164
PS9, Line 164:     synchronized (thriftDb_) { return thriftDb_.getDb_name(); }
Now getting the name is synchronized on an HMS call. That can't be good for performance. The planner/analyzer gets the name all the time, so we really don't want to do this.

The name should be invariant or all heck breaks loose in the planner. So, if we swap out Thrift objects, pull the name outside the thrift object and cache it in a final field.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@50
PS9, Line 50:  * in Hive metastore using the the public API <code>get_next_notification</code> These events could
Impala generally does not use Javadoc markup, sadly. But, if you want to use it (we really should), then a reference to a Java method should be: {@link Class#method}.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@63
PS9, Line 63:  *   <li>CREATE_TABLE, CREATE_DATABASE</li>
Wrong form for HTML lists. I think you want a definition list:

<dl>
  <dt>CREATE_TABLE, CREATE_DATABASE<dt>
  <dd>A new table/database is ... </dd>
...

That sad, if Vuk was here, he'd have you rip all this out. So, a broader question is: do we want our Javadoc to follow C++ doc standard so the C++ folks can follow it, or Javadoc standard so the documentation looks correct when formatted? I prefer the Javadoc approach, so would encourage you to keep the (corrected) formatting.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@91
PS9, Line 91:   private static final Logger LOG = Logger.getLogger(MetastoreEventsProcessor.class);
Statics generally go before instance members. Move this up with the constant string.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@106
PS9, Line 106:       ExtendedJSONMessageFactory.getInstance();
Same as above, and below.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@118
PS9, Line 118:     // know  if we need to ignore some of events which are already applied in catalogD
To avoid race conditions, separate creation from starting. Have a separate method that starts the timer, to be called once the system is ready for action.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@137
PS9, Line 137:   void scheduleAtFixedDelayRate(long pollingFrequencyInSec) {
Please put explicit public/private/protected declarations before each class and method so we can determine your intent. Shouldn't this one be private? Or, if we make the race-avoiding change above, it would be public, called once.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@145
PS9, Line 145:         LOG.warn(String.format(
LOG.error?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@158
PS9, Line 158:   void processHMSNotificationEvents() throws ImpalaException {
Please document the timer semantics supported here.

This is not synchronized. Say we poll every 5 seconds. Say we get a very slow HMS message and the previous poll is still in progress when the next tick occurs. Would we get two poll events overlapping? Or, will the timer skip a tick if the previous has not yet completed? That is, does the scheduler handle this case, or must you handle it here?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@177
PS9, Line 177:       throw new MetastoreNotificationException(
To whom are we throwing this? This method was called by the scheduler, wasn't it? Does the scheduler know what to do with this?

Or, should we log the error here?

What happens if the errors continue? (HMS server was shut down on host x and moved to host y. It is never coming back up.) Should we retry for x times, then turn off polling? Use an exponential back-off?

Whatever we do, please document the error handling strategy.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@186
PS9, Line 186:             .format("Unable to process notification event %d due to %s. Ignoring this event...",
If we get this in an escalation, we'll need to debug. At the very least, dump some info for the kind of event. Ideally dump things like table name, type of change, etc.

Next, document why it is OK to ignore an event and continue. Since we are invalidating, the worst that will happen is that something is not invalidated that should be.

But, if we ever wanted to replay events, failing on one would be fatal, continuing would corrupt the metadata. A fall-back there would be a global invalidate for the DB in question.

Should we use that as the fall-back here?

Given the severity, this should be LOG.error.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@198
PS9, Line 198:     LOG.info("Processing event " + event.getEventId() + " : " + event.getEventType()
Use this info in the error message above. In a production server, the info events may be turned off, we'll get only the error/warn events.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@202
PS9, Line 202:     Db db = catalog_.getDb(dbName);
Are there any events for which the DB name will be missing? System-wide events?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@204
PS9, Line 204:     switch (event.getEventType()) {
Is the event type provided as an enum or just a string? String comparisons work, are slower than enum/int comparisons.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@209
PS9, Line 209:         invalidateCatalogTable(table);
No need to invalidate the entire table (potentially 1000s of partitions) because one changed. Talk to Todd to see if he can recommend a more focused invalidation.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@212
PS9, Line 212:         catalog_.addTable(dbName, tblName);
No need to clutter up the catalog with new tables that no query has requested. Maybe just ignore these events. We'll get the table data on the first query for that table.

This is particularly true for temp tables and other transient items.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@221
PS9, Line 221:           invalidateCatalogTable(table);
Should the fall-back logic be implemented inside the rename table event method?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@225
PS9, Line 225:         catalog_.removeTable(dbName, tblName);
This one prompts questions about synchronization. Suppose 10 tables are removed. Suppose we're planning a query at the same time. The way this works, for each removal, we'll lock the catalog, make the change, and unlock. A planning session will see multiple changes. The current logic will (I believe) fail the plan and restart.

Since we do the 10 removes (or other changes) one-by-one, it means we will fail and restart planning for any in-flight queries once for each change (10 times here) rather than once for overall poll.

Do we like this? Does this help with the goal of performance? Can we do something else?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@235
PS9, Line 235:         catalog_.addDb(dbName, msDb);
Don't add the DB for the same reason cited for tables. If this Impala is used for Sales, we really don't need to take space in the cache for DBs created by Production, say.

Also, if this is a remove, we should remove the old DB, then add a new one. Does this avoid the need for the db Thrift switch code discussed in the DB class?

Reason: we need to remove all tables for the DB, and all partitions for all those tables. If a DB is dropped and replaced, none of the prior tables survive. We should not retain them in our cache.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@238
PS9, Line 238:         if (db != null) {
Let's think a bit. We're checking for null db on every event. We don't care about a new DB. So, should we do the "we don't have the DB in our cache so we don't care about this event" check before the switch statement?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@261
PS9, Line 261:         LOG.warn(
We'll get thousands of these. Set this to Log.detail/debug (whatever the finest-grain level is). We know it is OK to ignore them because you wrote the code stating that this is OK, not much the user can do with the warnings other than call for help.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@271
PS9, Line 271:    * table in-place
We're just invalidating, not renaming. So, just remove the old table. Impala will load the new one on demand when the next query arrives that uses it.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@281
PS9, Line 281:             event.getMessage());
Can deserialization fail? If so, what does that exception due to the scheduler when propagated back up through the event thread?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@290
PS9, Line 290:         return false;
Wait, then why did we receive this event? Did something else about the table change?

Should we just invalidate the table to be safe if we don't know what changed?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@296
PS9, Line 296:       catalog_.addTable(newTable.getDbName(), newTable.getTableName());
Skip this step for reasons given above.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@299
PS9, Line 299:       throw new MetastoreNotificationException(e);
How is this handled?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@304
PS9, Line 304:    * Deserializes the event message and create a database object out of it.
nit: Deserializes ... create[s]...


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@315
PS9, Line 315:       if ("CREATE_DATABASE".equalsIgnoreCase(event.getEventType())) {
The switch statement already parsed out the event type. Looks like we're doing it here again so we can share some code at the top and bottom.

Better: one method per event. Put common code into methods, called from the handler methods.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@318
PS9, Line 318:                 .getCreateDatabaseMessage(event.getMessage());
This code block appears multiple times. For each I'll ask error handling strategy. Cries out for being moved to its own method. Each returns a different class. This can be handled with Java generics and passing the class of the expected message type.

Which suggests that the method either check the message class type before casting, or handle the class cast exception. This will make the code more resilient to unexpected changes to the RPC protocol since the protocol itself is not versioned (I believe.)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@326
PS9, Line 326:         if (!checkSupportedCasesForAlterDatabaseEvent(msDbBefore, msDb)) {
Good error handling. Can be made better. Change the method to "validateAlterDatabaseEvent". Have it return void. In the method, parse out each error case:

if (first problem occurs) {
  throw Something("describing the first problem"
} else if (second problem occurs) { ...

That way, we'll log the precise failure mode making it easier to diagnose issues in the field when all we have are logs.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@343
PS9, Line 343:       throw new MetastoreNotificationException(e);
Tests should trigger each error condition to see how they are reported in the log. Determine if we could track down the problem from the log messages. Error injection is helpful in such cases. Do we have a preferred mechanism? Else, we can invent something.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@365
PS9, Line 365:     if (table == null) {
Why would we call this for a null table? Preconditions instead?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@373
PS9, Line 373:         + " invalidated because the table was recreated in metastore");
Is this method only for recreation? Looks like it is called for ALTER_TABLE, which is not recreation. Better log message?

Also, how the heck does the catalog know if the table was removed or added? Is this the right method to be calling here? Does that method call out to HMS to learn about the table? If so, should we just invalidate instead?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@401
PS9, Line 401:   static class MetastoreNotificationException extends ImpalaException {
You should have gotten a warning here about lack of a serialization ID, I you are using an IDE. To solve that, add the suppress warnings annotation that your IDE suggests.

Also, where do we expect to catch the exception? If inside this code only, then a nested class is fine. If outside, then moving the class to a separate file is a good idea.

Finally, nested classes generally should appear at the top of the file. Mark the class as public unless you really want to hide if from other modules.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 20:06:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#28). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationFetchException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
14 files changed, 2,553 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/28
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 28
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
8 files changed, 1,280 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/10
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
10 files changed, 1,367 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/12
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 12
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1791/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jan 2019 01:05:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 28:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 28
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jan 2019 00:53:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 15:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@345
PS15, Line 345:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jan 2019 23:41:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................

IMPALA-7970 : Add support for automatic invalidates by polling metastore events

This change adds support to CatalogD to poll metastore events to issue invalidate
on tables automatically. It adds basic infrastructure to poll HMS notifications events
at a configurable frequency using a backend config called hms_event_polling_frequency_s
flag. Currently, it issues invalidate at tables when it received alter events on table
and partitions. It also adds tables/databases and removes tables from catalogD when it
receives create_table/create_database and drop_table/drop_database events. The default
value of hms_event_polling_frequency_s is 0 which disables the feature. A non-zero
value in seconds of this configuration can be used to enable the feature and set
the polling frequency.

The change also adds a new test class to test the basic functionality for each of the
event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
9 files changed, 1,018 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/7
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,926 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/18
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 17:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/17/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/17/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@345
PS17, Line 345:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 17
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jan 2019 00:41:19 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 22:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@337
PS22, Line 337:                 + "invalidated since it does not exist anymore", getFullyQualifiedTblName()));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/22/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@443
PS22, Line 443:       if (eventsProcessor_.getStatus() != EventProcessorStatus.ACTIVE) eventsProcessor_.start();
line too long (96 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jan 2019 21:47:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 9:

(20 comments)

Remaining comments, mostly about the tests.

High level, seems the open questions are:

* That bit about replacing the guts of the DB object.
* Error handling and reporting
* Whether the event handler should attempt to partially load DBs and tables, or just do an invalidate.
* Invalidation of specific partitions rather than all of them.

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@64
PS9, Line 64:   private static MetastoreEventsProcessor eventsProcessor;
Not thrilled with the convention, but the team likes an underscore after each field name.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@109
PS9, Line 109:     assertNotNull(catalog.getDb(TEST_DB_NAME));
Per prior review comments, should not add the new DB.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@120
PS9, Line 120:     createDatabase("database_to_be_dropped");
If we don't add the DB automatically, should reference it and assert it is in the cache.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@122
PS9, Line 122:     assertTrue(catalog.getDb("database_to_be_dropped") != null);
assertNotNull(...)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@125
PS9, Line 125:     assertTrue("Database should not be found after processing drop_database event",
assertNull(...)

Oddly, later tests do use assertNull(). So, modify the code here, and the other cases where assertTrue is used to check for null/not null.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@136
PS9, Line 136:   @Test(expected = DatabaseNotFoundException.class)
This is confusing. If we expect an exception, then the exception terminates the execution. Logically, it should come from the last statement in the body. But, the last statement is a Unit assertion. So, something is strange.

Often easier to do this:

try {
  doSomethingThatShoudFail();
  fail();
} catch (ExpectedException e) {
  // Expected
}


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@139
PS9, Line 139:     final String TBL_TO_BE_DROPPED = "tbl_to_be_dropped";
Looks like a constant. Move outside the method and make static.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@148
PS9, Line 148:     loadTable(TBL_TO_BE_DROPPED);
Does this assert that the table exists in HMS, was loaded in the cache, and has the expected contents? Else, when we drop it, we can't tell if the drop code is broken if we don't know if the table was actually in the cache.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@155
PS9, Line 155:     // throws DatabaseNotFoundException
If it throws, we won't get to the assertTrue.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@161
PS9, Line 161:   @Ignore("Disabled until we fix Hive bug to deserialize alter_database event messages")
Reference HIVE JIRA ticket number.

Also, should reference this bug in the event processor code.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@217
PS9, Line 217:     createTable(TEST_TABLE_NAME_NONPARTITIONED, false);
Start by asserting that the table is not yet in the cache?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@255
PS9, Line 255:     List<List<String>> partVals = new ArrayList<>(1);
Allocating just 1 member up front, then adding four items. Maybe just omit the initial size arg; not helping us much here.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@277
PS9, Line 277:     // change some partitions
There is a recent fix that does specialized calls to HDFS to check for updated vs. new partitions. It will be tricky to combine that test with this one, but we should ensure that the automated way of updating a partition uses the call optimizations from that fix.

I already noted in previous comments that we might not be using that path.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@296
PS9, Line 296:     eventsProcessor.processHMSNotificationEvents();
Any race conditions here? Are the events immediately available after the mutation call? Will all events come in a single message?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@299
PS9, Line 299:         "Old named table is still existing", catalog.getTable(TEST_DB_NAME, "old_name"));
Nit: is still existing --> still exists


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@307
PS9, Line 307:     // Hive does not seem to create the events when table parameters are updated
Do we need them? Hive JIRA ticket number?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@311
PS9, Line 311:     assertTrue("Table should be incomplete after alter table add parameter",
Seems we need more granularity here.

The catalog needs a method to report if a table is cached, used just for testing. If an event causes the table to become invalidated, then the best way to tell is to check that the table is no longer cached.

Then, we should retrieve the metadata in the normal way to ensure that the update was, in fact applied. This second path mimics what Impala will actually do and validates that the goal (Impala sees the change) did occur.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@320
PS9, Line 320:         catalog.getTable(TEST_DB_NAME, TEST_TABLE_NAME_NONPARTITIONED)
Again, load table, verify that the new column appears.


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@331
PS9, Line 331:     // check invalidate after alter table remove column type
remove column *type*? Or, remove column?


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@430
PS9, Line 430:   private void alterTableRename(String tblName, String newTblName) throws TException {
Beyond the scope of this project, but the fact that you had to write these suggests we could do some refactoring. Impala already does these operations to handle DDL statements. We should not have to code up the same stanzas here. A future improvement is to create an Impala-specific client that wraps the Hive client and handles the boilerplate.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 21:12:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 7:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/1659/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 00:53:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 8:

(46 comments)

Posting my initial set of comments. Haven't looked at the tests yet. Will take another pass after code refactoring. (Also, please update your profile name in gerrit settings).

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG@7
PS8, Line 7: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
nit: wrap lines to less than 72 chars.


http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG@21
PS8, Line 21: 
Is the self-event detection coming in a separate patch?


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:

http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@246
PS8, Line 246: invalidate or refresh
This only invalidates right? (refresh has a different meaning in Impala's context)


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@247
PS8, Line 247: cached metadata about tables
nit: rephrase to  ...cached table metadata based on...


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@251
PS8, Line 251:  
..has..


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@253
PS8, Line 253:     "Additionally, "
Since this is a user-facing message, we should probably clarify that this is for "external" events.


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@258
PS8, Line 258: To enable this feature, a "
             :     "non-zero "
             :     "value of the flag must be applied to catalogd and impalad.
Looks like this is already mentioned above. Also, is it needed on the coordinators? Doesn't look like it.


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@260
PS8, Line 260:     "value of the flag must be applied to catalogd and impalad.");
nit: format to fewer lines like others.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@229
PS8, Line 229:   private MetastoreEventsProcessor metastoreEventProcessor;
Add a oneliner comment. Also, we use an "_" prefix for class variables. foo_


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@292
PS8, Line 292: if configured
nit: sounds vague, rephrase?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@292
PS8, Line 292: In order to determine if
             :    * event polling is enabled, this method looks at the value of HMS polling frequency in
             :    * <code>BackendConfig</code>.
I think we can get rid of the implementation detail here, remove?

Also, I don't think you are doing that check here? Return early (and log) if it is disabled?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@304
PS8, Line 304: LOG.error(
             :             "Unable to fetch the current notification event id from metastore."
             :                 + "Metastore event processing will be disabled.");
Isn't this a fatal error? (Include the stack trace in the error log)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@45
PS8, Line 45: events
Probably good to preface this with a what an "event" is with a pointer to Hive's class. Basically when is an event generated etc (since that isn't quite obvious for Impala folks). Also probably worth mentioning that the goal is to apply changes external to Impala.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@47
PS8, Line 47: The polling of the events is from a
            :  * given start event id at the construction time. Subsequently, it updates the last event
            :  * id until which it has already synced so that next poll is from the last synced event
            :  * id.
Maybe just rephrase to "we keep track of last synced event in each iteration..."


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51
PS8, Line 51: 1000
MAX_EVENTS....


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51
PS8, Line 51: In case there is a use-case in the future, we can change the batch size to a
            :  * configurable value. The polling frequency can be configured using the backend config
            :  * hms_event_polling_frequency_s
Remove?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@56
PS8, Line 56:  * metastore. In case of CREATE_TABLE/CREATE_DATABASE events, a new table/database is
nit: Make this bullet points instead? (supported / non-supported)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@64
PS8, Line 64: In order to function correctly, it assumes that Hive metastore is configured correctly
            :  * to generate the events. Following configurations should be set in metastore
            :  * hive-site.xml so that events have sufficient information and actions can be taken base
            :  * on them.
            :  *
            :  * <code>hive.metastore.notifications.add.thrift.objects</code> should be set to
            :  * <code>true</code> so that event messages contain thrift object and can be used to
            :  * create new objects in Catalog <code>hive.metastore.alter.notifications.basic</code>
            :  * should be set to <code>true</code> so that all the needed ALTER events are generated
I think this is implementation detail can be omitted in the class comment (since you already mentioned it in the init methods).


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@84
PS8, Line 84: METASTORE_NOTIFICATIONS_ADD_THRIFT_OBJECTS
_CONFIG_KEY


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@84
PS8, Line 84:   static final String METASTORE_NOTIFICATIONS_ADD_THRIFT_OBJECTS =
for consistency, use public/private?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@88
PS8, Line 88:   private long lastSyncedEventId_;
Observability: expose this as a catalog server metric? (look at <catalog-webui>/metrics). Also helps with testing.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@93
PS8, Line 93:   private final ScheduledExecutorService scheduler =
nit: "_" for class members (multiple places)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@109
PS8, Line 109:   private static final int EVENTS_BATCH_SIZE = 1000;
oneliner comment.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@113
PS8, Line 113: Preconditions.checkNotNull(catalog);
             :     this.catalog_ = catalog;
this.catalog_ = Preconditions.checkNotNull(catalog);


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@122
PS8, Line 122: if (pollingFrequencyInSec > 0) {
             :       scheduleAtFixedDelayRate(pollingFrequencyInSec);
             :     }
Probably worth moving this check to the parent class and skip the c'tor if this is not set.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@128
PS8, Line 128: of
update.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@139
PS8, Line 139: Starting metastore event polling with interval " + pollingFrequencyInSec
             :         + " seconds."
nit: String.format()


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@147
PS8, Line 147: );
include stack trace


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@156
PS8, Line 156:   private long getCurrentNotificationID() {
Looks like the only caller is processHMSNotificationEvents(), can we reuse the msClient? (do we need a separate method? can't we just move it to the try-catch block inside processHMSNotificationEvents()?)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@174
PS8, Line 174: lastSyncedEventId_ =
             :         lastSyncedEventId_ == -1 ? getCurrentNotificationID() : lastSyncedEventId_;
             :     if (lastSyncedEventId_ == -1) {
             :       LOG.warn(
             :           "Unable to fetch current notification event id. Cannot sync with metastore");
             :       
Like I commented below, it appears to me that  (lastSyncedEventId_ != -1) is a precondition. Thoughts?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@182
PS8, Line 182: CurrentNotificationEventId currentNotificationEventId =
             :           msClient.getHiveClient().getCurrentNotificationEventId();
             :       long currentEventId = currentNotificationEventId.getEventId();
             :       if (currentEventId > lastSyncedEventId_) {
Can we skip this and directly do a single RPC to fetch the latest events?

events = getEvents();
if (events.size() == 0) break;


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@187
PS8, Line 187: EVENTS_BATCH_SIZE
Looks like we are only doing only a single batch per iteration. Should we rename this to MAX_EVENTS_PER_RPC or something?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@191
PS8, Line 191:           // update the lastSyncedEventId before processing the event itself
Looks like this event is lost forever. Any idea how this can be surfaced back to the user?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@196
PS8, Line 196:           processEvent(event);
If a single event processing throws an exception, we seem to be breaking out of the loop. Does that mean everything after that is lost too?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@212
PS8, Line 212: dbName
This seems to be optional in the thrift spec and the next line there is a (db != null) check. Are there events where this is not set in the response? Should we add those tests?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@215
PS8, Line 215:       case "ADD_PARTITION":
Strings look risky, is there no EventType enum or some such?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@221
PS8, Line 221:       case "CREATE_TABLE":
This seems to contain too much code and branches. Can we define an EventHandler interface (or abstract class) and implement it for various Events and register them at init? That simplifies the code here to dispatch. Thoughts?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@222
PS8, Line 222:         catalog_.addTable(dbName, tblName);
What if this races with an addTable() from the local cluster?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@235
PS8, Line 235:         catalog_.removeTable(dbName, tblName);
Same as above. What if there are races? (multiple cases below)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@257
PS8, Line 257: /*if (db != null) {
             :           msDb = MetastoreEventsProcessor.getDatabaseFromMessage(event);
             :           db.setMSDb(msDb);
             :         }*/
Is this intentional? (given you changed the Db class locking protocol)


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@287
PS8, Line 287:   private boolean processRenameTableEvent(NotificationEvent event)
It looks like to me like these should call into the CatalogOpExecutor (modulo apply*() calls) primarily for

- Locking considerations (races)
- To avoid code duplication


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@347
PS8, Line 347: Check if %s is set to true in metastore configuration
Lets force this check?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@401
PS8, Line 401: startSyncFromId
If startSyncFromId == -1, shouldn't that be a fatal error?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@402
PS8, Line 402:     if (INSTANCE == null) {
Do we need to validate the HiveClient config?

property>
  <name>hive.metastore.notifications.add.thrift.objects</name>
  <value>true</value>
</property>
<property>
  <name>hive.metastore.alter.notifications.basic</name>
  <value>false</value>
</property>


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@412
PS8, Line 412:   static class MetastoreNotificationException extends ImpalaException {
Move to its own class file


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/test/resources/postgresql-hive-site.xml.template
File fe/src/test/resources/postgresql-hive-site.xml.template:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/test/resources/postgresql-hive-site.xml.template@223
PS8, Line 223: <!-- This property is required to issue invalidates based on metastore events. See IMPALA-7954 for details -->
nit:wrap lines.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Dec 2018 02:23:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 14:

(5 comments)

I'm not planning on doing a full review since Bharath and Paul are on it. The concurrency aspects I looked at are much better in this version. I had some follow on questions about the interaction between the initial load of the catalog and the invalidation - I think you anticipated that to some extent.

If it's too much to address in this patch we could consider merging this without addressing that but keeping it behind an experimental feature flag until some more of the concerns have been addressed.

http://gerrit.cloudera.org:8080/#/c/12118/14/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:

http://gerrit.cloudera.org:8080/#/c/12118/14/be/src/common/global-flags.cc@245
PS14, Line 245: DEFINE_int32(hms_event_polling_frequency_s, 0,
Maybe make it hidden or mention that it's experimental in the text here if it's not ready for prime time yet?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@299
PS14, Line 299:    * Initializes Metastore event processor object if
This occurs before the initial metadata is loaded in reset(), right? So we might process some events that occurred before we got an initial snapshot of the data. I think this is probably OK if the events are idempotent, but seems worth mentioning in the comment.

Actually one question is how this interacts with reset(), either the initial reset() on startup or a later one from "INVALIDATE METADATA". What happens if events start to be processed when the catalog is empty before reset() loads all the data? And should we reset the current event ID when doing an invalidate?

I'm wondering if those questions lead us to resetting currentNotificationId in reset() and stopping/pausing the even processing until reset() completes. That would at least reduce the number of possible races.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@319
PS14, Line 319:       LOG.fatal("Unable to fetch the current notification event id from metastore."
This does actually terminate the process - it call to the glog LOG(FATAL) which calls abort(). I think we don't want this since it generates a core dump - Maybe LOG.error() + throwing an exception is the right behaviour.

We've had issues before with misconfigured catalogds generating a ton of core dumps.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1342
PS14, Line 1342: null
?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@169
PS14, Line 169:     // TODO figure out if CatalogD sync with HMS is in-process or completed so that we
Ah, I think this is referring to the same kind of issue that I mentioned earlier with reset() running in parallel with this thread.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jan 2019 00:03:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java@112
PS8, Line 112:     synchronized (thriftDb_) { thriftDb_.setMetastore_db(msDb); }
> I see your point. Initially, I wanted to model the changes to metastore db 
The locking scheme does require the cooperation of multiple classes so method access modifiers aren't likely to be sufficient to prevent other classes from doing bad things (it's good if we make fewer things public for sure though).

I think the precedent to look at here are the various alterDatabase operations in CatalogOpExecutor. Those follow the protocol described in the CatalogOpExecutor class comment. In those cases metastoreDdlLock_ is the lock that's preventing concurrent updates to thriftDb_.

I was trying to understand if anything prevents concurrent reads + writes to thriftDb_ though and I don't think there currently is anything, so it seems like some anomalies could result from that. There's also some logic in CatalogOpExecutor that mutates thriftDb_ in-place.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 20:11:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 8:

> Uploaded patch set 8.

Patch set 8 adds the missing Apache license headers.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 02:18:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
10 files changed, 1,375 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/13
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 9:

(34 comments)

Latest patch works through some of the comments. Still working through the rest of the comments

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG@7
PS8, Line 7: IMPALA-7970 : Add support for metastore event based automatic invalidate
> nit: wrap lines to less than 72 chars.
done


http://gerrit.cloudera.org:8080/#/c/12118/8//COMMIT_MSG@21
PS8, Line 21: The change also adds a new test class to test the basic functionality
> Is the self-event detection coming in a separate patch?
Yes, self-event detection is still a work in progress and would be done as part of IMPALA-7972


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:

http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@246
PS8, Line 246: invalidate cached tab
> This only invalidates right? (refresh has a different meaning in Impala's c
Right now, its only invalidate but I plan to make use of refresh in IMPALA-7973. I will change it to invalidate since that is the current state of the feature.


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@247
PS8, Line 247: These metastore events could
> nit: rephrase to  ...cached table metadata based on...
done


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@251
PS8, Line 251: \
> ..has..
done


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@253
PS8, Line 253:     "creation or removal of objects from metastore, catalogd adds or removes such "
> Since this is a user-facing message, we should probably clarify that this i
The first line of the message mentions metastore events. Do you think that is not clear to a user? Added one line which describes how the metastore events could be generated from external systems like Apache Hive or another instance of Impala cluster talking with the same metastore.


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@260
PS8, Line 260: 
> nit: format to fewer lines like others.
I had formatted it like the others, but the clang-format-diff script which I use to format the patch doesn't seem to like it. Any idea how can fix it automatically?


http://gerrit.cloudera.org:8080/#/c/12118/8/be/src/common/global-flags.cc@258
PS8, Line 258: r the old generation to be almost "
             :     "full and trigger an invalidation on recently unused tables");
             : 
> Looks like this is already mentioned above. Also, is it needed on the coord
removed the redundant line. Also, yes, not need on coordinators as of now.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@229
PS8, Line 229: 
> Add a oneliner comment. Also, we use an "_" prefix for class variables. foo
done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@292
PS8, Line 292: 
> nit: sounds vague, rephrase?
added some more meat to this. Please suggest improvements if you think something more is needed in the text. Thanks!


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@292
PS8, Line 292: 
             :   /**
             :    * Initializes Metastore event
> I think we can get rid of the implementation detail here, remove?
Good point. Added check and return early logic.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@304
PS8, Line 304:    + "interval is %d", eventPollingInterval));
             :       return;
             :     }
> Isn't this a fatal error? (Include the stack trace in the error log)
done. Did you mean changing the log level to fatal as well?


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/Db.java@112
PS8, Line 112:     synchronized (thriftDb_) { thriftDb_.setMetastore_db(msDb); }
> This additional ad-hoc locking without explanation of how it integrates wit
I see your point. Initially, I wanted to model the changes to metastore db object similar to how functions are added to its parameters. I noticed that Db class has public method to add a function which can modify the state of the metastoredb object parameters without taking the version lock. However, as I look at the its callers, I see that this method is used from CatalogServiceCatalog class which takes a version lock. Perhaps the methods in Db class which modify the hms database parameters should not be made public in such a case. I will make changes to the patch so that updates to metastore db object are done before taking version lock and incrementing the version number


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@45
PS8, Line 45: 
> Probably good to preface this with a what an "event" is with a pointer to H
done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@47
PS8, Line 47: us DDL operations like create/alter/drop
            :  * operations on database, table, partition etc. Each event has a unique incremental id and the
            :  * generated events are be fetched from Metastore to get incremental updates to the metadata stored
            :  * in 
> Maybe just rephrase to "we keep track of last synced event in each iteratio
done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51
PS8, Line 51: be g
> MAX_EVENTS....
done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51
PS8, Line 51: erated by external Metastore clients like Apache Hive or Apache Spark as well as other
            :  * Impala clusters configured to talk with the same metastore.
            :  *
> Remove?
removed


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@56
PS8, Line 56:  * each event type. We keep track of the last synced event id in each polling iteration so the next
> nit: Make this bullet points instead? (supported / non-supported)
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@64
PS8, Line 64:   A new table/database is created in Catalog respectively. The newly created table/database are
            :  *   Incomplete and should get loaded lazily when needed
            :  *   <li>DROP_TABLE, DROP_DATABASE</li>
            :  *   The table/database is dropped from catalog if they are present
            :  *   <li>ALTER_TABLE</li>
            :  *   In case of alter table event, currently the code issues a invalidate table command. There is a
            :  *   special case of this event in case of renames, where the old table is removed and a new
            :  *   IncompleteTable with the new name is created.
            :  *   <li>ALTER_DATABASE</li>
> I think this is implementation detail can be omitted in the class comment (
removed this part of javadoc as suggested


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@84
PS8, Line 84:  String METASTORE_NOTIFICATIONS_ADD_THRIFT
> _CONFIG_KEY
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@93
PS8, Line 93:   private final ScheduledExecutorService scheduler_ =
> nit: "_" for class members (multiple places)
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@109
PS8, Line 109:   // maximum number of events to poll in each RPC
> oneliner comment.
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@113
PS8, Line 113:   CatalogServiceCatalog catalog, long startSyncFromId, long pollingFrequencyInSec) {
             :     this.catalog_ = Precondi
> this.catalog_ = Preconditions.checkNotNull(catalog);
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@128
PS8, Line 128: at
> update.
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@139
PS8, Line 139: tring.format("Starting metastore event polling with interval %d seconds.",
             :         pollingFreque
> nit: String.format()
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@147
PS8, Line 147: , 
> include stack trace
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@156
PS8, Line 156:    */
> Looks like the only caller is processHMSNotificationEvents(), can we reuse 
This is not needed. Removed this method.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@174
PS8, Line 174:   LOG.info(String.format("Received %d events. Start event id : %d",
             :           response.getEvents().size(), lastSyncedEventId_));
             :     } catch (TException e) {
             :       throw new MetastoreNotificationException(
             :           "Unable to fetch notifications from metastore. Last synced event id is "
             :       
> Like I commented below, it appears to me that  (lastSyncedEventId_ != -1) i
This check is redundant and not required. Removed it.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@182
PS8, Line 182: try {
             :         processEvent(event);
             :       } catch (MetastoreNotificationException ex) {
             :         LOG.warn(String
> Can we skip this and directly do a single RPC to fetch the latest events?
The getCurrentNotificationEventId is much cheaper (and hence faster too) than fetching all the events on the HMS server side. I haven't profiled it but I think in the common case of not all the polling iterations detect new events, this implementation is faster than fetching events and making sure that events.size() > 0. I will add a comment since this is not obvious to the reader


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@187
PS8, Line 187: (), ex.getMessage
> Looks like we are only doing only a single batch per iteration. Should we r
Done


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@191
PS8, Line 191:         // throwing exception until catalogD is restarted
> Looks like this event is lost forever. Any idea how this can be surfaced ba
This would happen in a rare case of a bug in the event serialization and deserialization or a misconfiguration on the HMS side. The processEvent throws a exception and it is logged as a error. I am not sure if there are better ways to surface this to user. I would be happy to implement if you have better suggestions.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@196
PS8, Line 196: 
> If a single event processing throws an exception, we seem to be breaking ou
They are not lost forever since the lastSycnedEventId is only updated one by one and we will request for these events again in the next batch. But yes, I think we can improve this to process to attempt to process the rest of the events in the batch. Fixed it.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@212
PS8, Line 212: bName,
> This seems to be optional in the thrift spec and the next line there is a (
I am not sure I understood this comment correctly. The db!=null check makes sure that catalogD has this db object since we rely on that to get the table object from catalog. The tblName could be null in the events like CREATE_DATABASE


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@215
PS8, Line 215:       case "ALTER_TABLE":
> Strings look risky, is there no EventType enum or some such?
The eventTypes from the metastore API are strings. We can create our own enums which map to these strings if you think that is useful.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 00:58:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 18:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1805/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 21:28:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 27: Code-Review+2

Thanks for patiently addressing all the review comments. Nice first contribution to the project :-). I can kick-off a GVO once you fix the line overflows.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 27
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 18:19:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,926 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/19
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#17). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,900 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/17
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 17
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 29:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3681/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 29
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 21:27:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 16:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12118/16/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/16/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@20
PS16, Line 20: import static org.apache.impala.catalog.events.MetastoreEventsProcessor.METASTORE_NOTIFICATIONS_ADD_THRIFT_OBJECTS_CONFIG_KEY;
line too long (126 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/16/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@347
PS16, Line 347:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jan 2019 00:34:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 19:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@364
PS19, Line 364:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 21:10:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1792/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 17
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jan 2019 01:34:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 6:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/1658/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 00:47:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 21:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@364
PS21, Line 364:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 01:11:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,886 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/15
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 25:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@426
PS25, Line 426:                 + "invalidated since it does not exist anymore", getFullyQualifiedTblName()));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@481
PS25, Line 481:       if (eventsProcessor_.getStatus() != EventProcessorStatus.ACTIVE) eventsProcessor_.start();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@725
PS25, Line 725:   private void createDatabaseFromImpala(String dbName, String desc) throws ImpalaException {
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@751
PS25, Line 751:   private void createTableFromImpala(String tblName, boolean isPartitioned) throws ImpalaException {
line too long (100 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 25
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 26 Jan 2019 01:41:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1691/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 10
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Dec 2018 02:15:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 22:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1847/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jan 2019 22:41:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 18:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/18/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/18/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@364
PS18, Line 364:                     + "table event. Check if %s is set to true in metastore configuration",
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 18
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 21:06:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 25:

(12 comments)

lgtm. Final round of comments from my side. Given the state the patch is in, I think we should get the patch into the master branch which makes it easier for concurrency testing. Also since all of this code is behind a feature flag, it should not affect any existing deployments.

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@308
PS25, Line 308:       LOG.info(String.format("Metastore event processing is disabled. Event polling "
nit: fix formatting.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1833
PS25, Line 1833: if (db == null) return null;
               :       if (!db.containsTable(tblName)) return null
nit: merge into a single stmt.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@145
PS25, Line 145: no
not


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@149
PS25, Line 149:      * removed a table, the create event received will try to add the object again.
This probably needs some thought during concurrency testing.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@166
PS25, Line 166: drop event later
Do we also need to remove the drop event or are we just letting it fail silently?


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@174
PS25, Line 174:         fromIndex++;
Log how many events got filtered out? Helps diagnosing incase there are any bugs in this area.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@356
PS25, Line 356: if (this.dbName_.equalsIgnoreCase(dropTableEvent.dbName_) && this.tblName_
              :               .equalsIgnoreCase(dropTableEvent.tblName_)) {
              :             return true;
              :           }
nit: simplify to return dbName_.equals() && tblName_.equals();


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@360
PS25, Line 360: else if (event.eventType_.equals(MetastoreEventType.ALTER_TABLE)) {
              :           // if this create table was renamed to some other name in the future return true
              :           AlterTableEvent alterTableEvent = (AlterTableEvent) event;
              :           if (alterTableEvent.isRename_ && this.dbName_
              :               .equalsIgnoreCase(alterTableEvent.tableBefore_.getDbName()) && this.tblName_
              :               .equalsIgnoreCase(alterTableEvent.tableBefore_.getTableName())) {
              :             return true;
              :           }
clarify why rename qualifies as an inverse op for create?


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@295
PS25, Line 295: LOG.info(String
              :           .format("Received %d events. Start event id : %d", response.getEvents().size(),
              :               lastSyncedEventId_));
nit: format to fewer lines.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@319
PS25, Line 319:   LOG.warn(String.format(
              :             "Not processing notification events since event processing status "
              :                 + "is %s. Last synced event id is %d", currentStatus,
              :             lastSyncedEventId_));
nit: format to fewer lines.


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@339
PS25, Line 339: 
              :     // remove some partitions
              :     // change some partitions
?


http://gerrit.cloudera.org:8080/#/c/12118/25/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@583
PS25, Line 583:  in
?



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 25
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Jan 2019 23:23:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 15:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/1790/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Jan 2019 00:14:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 25:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1897/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 25
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 26 Jan 2019 02:17:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 29: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 29
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jan 2019 03:32:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 28: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 28
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 21:26:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 14:

(21 comments)

Nice improvements. Comments mostly relate to ways to simplify the processing logic. Feel free to ping me directly to discuss.

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@293
PS14, Line 293:     if (metastoreEventProcessor_ != null) {
The var != null pattern is error prone. Would be great to have the MetastoreEventsProcessor be an interface with two implementations: a null implementation (disabled) and a live implementation. Then, you just call the various methods without a check for null or not.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@325
PS14, Line 325:     }
Can this move to a static factory method on the MetastoreEventsProcessor implementation? Doing so puts all the logic for that abstraction in one place. All this method would do here is choose between the dummy and live versions.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1292
PS14, Line 1292:         return addDb(dbName, msDb) != null;
In master, addDb() returns a DB. This method returns a boolean. Is something amiss here?

Is this really an "addOrGetDb"? Do you want the DB object -- either an existing one or a new one? Else, if you call this, get true, then look up the DB, is there a race condition?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1344
PS14, Line 1344:   public Table addTableIfNotExists(String dbName, String tblName,
Checked. We do have existing methods called getOrLoadTable(). So, calling this one getOrAddTable might be more consistent.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1345
PS14, Line 1345:       Reference<Boolean> dbWasFound, Reference<Boolean> tblWasFound) {
This is rather an odd C++ way to do things. A more typical Java way is to declare a class that will return the required values.. Or, you can declare a Pair<Table,TableStatus> with TableStatus an enum of {CREATED, ALREADY_EXISTS, DB_NOT_FOUND}. The same enum could be returned for other add-or-get operations.

Pair<Table,AddOrGetStatus> addOrGetTable(String dbName, String tblName) ...


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1359
PS14, Line 1359:       } else {
The else is not necessary since prior block returned. Not a bug, just not necessary.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1365
PS14, Line 1365:       versionLock_.writeLock().unlock();
This is old school. We should change this so we can use try-with-resources:

try(LockInstance lock =versionLock.writeLock()) {
   ...
}

LockInstance would implement AutoCloseable to make this happen.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1462
PS14, Line 1462:       Reference<Boolean> tblWasfound, Reference<Boolean> tblMatched) {
Same issue with the references.

We already have a removeTable() method which is implemented to do a "remove table if exists". Is there commonality that should be exploited here?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1792
PS14, Line 1792:     versionLock_.writeLock().lock();
Race condition? We got the DB above before the lock. The DB could be invalid now that we have the lock.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1804
PS14, Line 1804:     return db.getTable(tblName);
The table was created above. We released the lock, then looked up the table again. Race conditions?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@52
PS14, Line 52:   public static final String ALTER_PARTITION_EVENT_TYPE = "ALTER_PARTITION";
Given that we use the event names not just to translate from Thrift event to handler, perhaps this should be an enum.

public EventType {
  CREATE_TABLE("CREATE_TABLE") ...
}

Then we can do:

  if (event == EventType.CREATE_TABLE) ...

The enum form is both faster and more robust.

Oh, and to get an enum value from a string you can define a static getEventType(String name) method to do the lookup.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@62
PS14, Line 62:   public static class MetastoreEventHandlerFactory {
I'm a bit confused by the structure. Looks like the event handlers are created once up-front. That means that the event is handled by a method that must take all its parameters via its "do something" method.

A more typical approach is to create an instance per event, with the instance holding all the required data. Indeed, if you do this, then you can pass the event object into your addOrGet methods which can fill in the "already exists", "just created", "table pointer" and other values.

The overhead of creating one more Java object is a small price to pay for the resulting simplicity.

Also, in this model, you don't need the EventType enum suggested above: the event instance is its own enum. There is no checking of "if the event is of type x" because each event has its own handler class.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@74
PS14, Line 74:     private final ConcurrentHashMap<String, MetastoreEventHandler> eventhandlers_ =
Also, if you use enums, each has an ordinal. You can then populate the table simply with a loop that populates an array:

MetaStoreEventHandler eventHandlers[] = new MetaStoreEventHandler[EventType.values().size].
for (int I = 0; I < EventType.values.size(); i++) {
  eventHandlers[I] = ...( EventType.getValue(i) );

The method names may be off; typed above from memory.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@87
PS14, Line 87:     public MetastoreEventHandler getOrCreate(String eventType) {
With the array, this becomes as simple as:

return eventHandler[ EventType.getEvent(eventType) ];


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@135
PS14, Line 135:     protected final Logger LOG = Logger.getLogger(this.getClass().getName());
Do you need a logger per class if they will all report the same class name? Or, do you want

  protected static final Logger LOG = Logger.getLogger(MetastoreEventHandler.class);


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@160
PS14, Line 160:         String expectedType, NotificationEvent event) {
This need not be a util method if we store the Db and Table name as members of each event object. Just call this to parse out the value and fill in the corresponding fields.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@269
PS14, Line 269:       JSONAlterTableMessage alterTableMessage =
This would be cleaner with the event-handler-per-event model. The factory would parse the event type, do the cast, an each handler would receive the subclass that it is designed to handle.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@49
PS14, Line 49:  * such events on the catalogD we can sync to external metadata operations by taking
Have the goals changed? Or, is this description jumping a few steps? Are we not keeping in sync by flushing the cache after each metastore change? We are not replaying the metastore changes onto our local cache?


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@53
PS14, Line 53:  *
Maybe put a <pre> ... </pre> around this to keep Eclipse and IntelliJ from treating this block as text when displaying JavaDoc.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@81
PS14, Line 81:  * . A object in catalog is stale if and only if its creationTime is <= creationTime of
Nit: move period to previous line.


http://gerrit.cloudera.org:8080/#/c/12118/14/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@117
PS14, Line 117:  * object will make sure that the catalog is stale before applying any such event.
Thanks for the very detailed explanation. Very helpful!



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 Jan 2019 00:59:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Reviewed-on: http://gerrit.cloudera.org:8080/12118
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationFetchException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
14 files changed, 2,553 insertions(+), 3 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 30
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
vihang@cloudera.com has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_frequency_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_frequency_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
A fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
M fe/src/test/resources/postgresql-hive-site.xml.template
9 files changed, 1,050 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/9
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@269
PS19, Line 269:     }
> I think fetching the event ID here can lead to a race where we miss a notif
Thanks for catching this. Fixed it the latest patch set. Although as discussed, there is still going to be small window of time where some of the events will be reprocessed. I have added a comment in CatalogServiceCatalog giving details about that case.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 00:17:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1731/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 14
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jan 2019 02:12:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20: Code-Review+1

(1 comment)

Thanks for addressing my comment. The aspects I looked at are good (I haven't reviewed the full thing)

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java@33
PS20, Line 33:    * Get the current event id. Useful for restarting the event processing from a event id
Maybe " on the HMS". Just to be clear that it's not the last event ID processed. Ok to ignore.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 00:33:19 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 29: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 29
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 21:27:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20:

(36 comments)

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@307
PS21, Line 307: BackendConfig.INSTANCE.getHMSPollingIntervalInSeconds()
> eventPollingInterval
Done


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@319
PS21, Line 319:  LOG.error("Unable to fetch the current notification event id from metastore."
              :               + "Metastore event processing will be disabled.",
              :           e);
> nit: Multiple places in the code, try to format in fewer lines. Something l
I used the command given in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868536 as advised by some of the other members of the team. Unfortunately, it doesn't like some of these lines.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@324
PS21, Line 324:           e);
> Will the code that handles this exception write to the log both the message
The LOG.error above logs the trace as well since it takes in the underlying cause exception as the second argument. The exception is uncaught in this particular case on the caller side, since we want CatalogD to not come up if it is configured to using event processing but for some reasons isn't able to instantiate one.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1206
PS21, Line 1206:     // Even though we get the current notification event id before stopping the event
> clarification: reset() is equivalent to invalidating everything and fetchin
This is similar to Tim's comment and our discussion. The line 1216 gets the currentEventID (latest) from HMS. And the event processing begins from that id. So the event processing jumps to the latest event id once the reset() is complete. There is still a slight chance that we will reprocess some of the events which are generated during reset() execution. It is better than missing those events which would lead to inconsistent state between catalog and HMS as far as event processing is concerned. The comment gives details of that case.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1216
PS21, Line 1216:     long currentEventId = metastoreEventProcessor_.getCurrentEventId();
> Should this be here? Or in the event processor itself? If here, why is it n
It needs to be here to avoid the race condition which can lead to possible missed events. We want to get the latestEventId before triggering the reset() so that we can then restart the event processing after reset from this eventId. If we move the code in the start() method, there is a chance that we will miss processing some of the events which happen during after reset is complete, but before we started the event processing. See Tim's comment in the earlier review comments which gives a nice example.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1360
PS21, Line 1360:       Reference<Boolean> dbWasFound, Reference<Boolean> tblWasFound) {
> I still think that using the reference is non-standard Java. A simple solut
used the second option of return boolean and throw exception when db is not found.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1363
PS21, Line 1363:     Db db = getDb(dbName);
> Let's think about this. We return the flags because we want to know if the 
if getting db before acquiring the lock has a race then it looks like addTable and renameTable also has a race. I will create separate JIRAs for fixing them


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1372
PS21, Line 1372:         tblWasFound.setRef(true);
> I wonder, do we really need to know if the table existed or not? Is it even
consider a case where user do create, drop, create with the same table name from Impala. In case of if the create event from the first create statement, the table presented in the event is stale and should not be used to add to the catalog. I think it is useful to log this information to make sure that the create event was ignored in such a case.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1475
PS21, Line 1475:       Reference<Boolean> tblWasfound, Reference<Boolean> tblMatched) {
> Same comments as above.
Moved the getDb() call in the block where lock is acquired first. I would like to defer the suggestion of changing the Reference to later patch in the interest of time.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1804
PS21, Line 1804:     Table incompleteTable = IncompleteTable.createUninitializedTable(db, tblName);
> move it after L1807?
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@130
PS20, Line 130:   public static abstract class MetastoreEvent {
> The new event class hierarchy looks very good.
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@138
PS20, Line 138:     protected final Logger LOG = Logger.getLogger(this.getClass());
> protected static final ... MetastoreEvent.class);
this.getClass() gives us the ability to log the implementation class name which is what I thought would be useful to debug. Else you will have to grep the statement in this class to locate the line of the code.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@160
PS20, Line 160:       if (!event.getEventType().equalsIgnoreCase("CREATE_DATABASE")
> Move this code to the constructor for the suggested MetastoreTableEvent cla
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@182
PS20, Line 182:     }
> Would move to the suggested MetastoreTableEvent class.
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@199
PS20, Line 199:     private final String tblName_;
> These two fields shadow fields of the same name in the base class. This is 
yeah, I missed cleaning this up in one of the many refactors this class went through.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@241
PS20, Line 241:         LOG.info(String.format("Added a table %s.%s after processing event "
> Nit: to make it easier to read/parse log records, maybe:
introduced a debugString method. Bharath had suggested the same.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@271
PS20, Line 271:         throw new MetastoreNotificationException("Unable to parse the alter table "
> This is a round-about way to do error handling. Generally, you EITHER
Would like to defer this to later in the interest of time.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@293
PS20, Line 293:           if (!invalidateCatalogTable(dbName_, tblName_)) {
> If this is a method, then no need to pass the fields since the method has v
missed that during refactor


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@318
PS20, Line 318:           throw new MetastoreNotificationException(String.format("Could not rename "
> Let's think about error handling a bit. There are two broad cases:
If the table rename did not happen, there is something very unexpected going on in the catalog. I think its better to halt the event processing in such a case. If the rename did not happen and the table in catalog still has the old name, it may be possible the future events act on the old name leading to inconsistent catalog state.

There is lot more to do in this feature and I would like to move on to the next sub-tasks. I would like to defer this suggestion to later unless you think it is critical for functionality of the patch.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@333
PS20, Line 333:         throw new MetastoreNotificationException(e);
> Suggestion: have MetastoreNotificationException take this class, or the mes
Good suggestion but in the interest of time the debugString method which does something similar is a good substitute for what your are suggesting here.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@155
PS21, Line 155: event %d of type %s on table %s
> I think it is cleaner to define a debugString(NotificationEvent event) {} t
Good suggestion. Added a util method which gets the debug string for a given list of message and arguments. It appends EventId and EventType information to the given message.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@160
PS21, Line 160: if (!event.getEventType().equalsIgnoreCase("CREATE_DATABASE")
              :           && !event.getEventType().equalsIgnoreCase("DROP_DATABASE")) {
              :         tblName_ = Preconditions.checkNotNull(event.getTableName());
              :       } else {
              :         tblName_ = null;
              :       }
> nit: Reformat to the following? (fewer branches and use enum)
The tblName_ is a final object so you can assign to it only once.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@181
PS21, Line 181: tblName_
> checkNotNull()?
Done


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@198
PS21, Line 198:    private final String dbName_;
              :     private final String tblName_;
> aren't these set in the parent c'tor?
yes, Looks like I missed removing it during one of the many iterations of the patch. Thanks for catching this.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@347
PS21, Line 347: droppedTable_
> isn't this more like tblTobeDropped_ ?
It depends on whether you see it as a object which is dropped in metastore or which needs to be dropped in Catalog. Since this object is retrieved by parsing the NotificationEvent and the type of the object is metastore.api.Table not impala.catalog.Table, I thought the name made sense.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@434
PS21, Line 434: //
> nit: remove, you are already in a comment block.
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@205
PS20, Line 205:       eventProcessorStatus_ = EventProcessorStatus.ACTIVE;
> Should we write something to the log, perhaps at DEBUG or TRACE levels, tha
added a info log.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@265
PS20, Line 265:       return metaStoreClient.getHiveClient().getCurrentNotificationEventId().getEventId();
> How is this used? How is it synchronized?
Initially this was part of start() method. But we needed to get the currentNotificationID from metastore in catalogServiceCatalog reset() method. So instead of duplicating the code, I created a method which is called from start() method.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@285
PS20, Line 285:       LOG.info(String.format("Metastore event processing restarted. Last synced event id "
> Isn't start() called only once? If not, what is the use case for pausing an
the start() method starts the event processing from whatever is the current event id in metastore. This is done when catalog service is started. start(eventId) resumes the event processing from the given eventId. THis is done from the reset() method. WHen a user issues invalidate metadata, the catalog state is cleared and we can jump to the latest eventId. Its also good to stop event processing during reset() to avoid races between events from reset()


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@360
PS20, Line 360:       Preconditions.checkState(lastProcessedEvent != null);
> The catch-block handles two types of exceptions. It also handles two source
good point. However, adding two catch blocks is not sufficient since event process can also throws MetastoreNotificationException. I added a simple check of lastProcessedEvent to find out if the getNextMetastoreEvents failed or the event.process()


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@371
PS20, Line 371:   private void dumpEventInfoOnErrors(NotificationEvent event) {
> The name indicates why the method might be called. But, the method itself d
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@373
PS20, Line 373:                             .append("Event id : " + event.getEventId() + "\n")
> Nit: If using a StringBuilder, write each field as a separate .append() cal
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@376
PS20, Line 376:                             .append("Database name : " + event.getDbName() + "\n");
> Another nit: colons usually have no leading space: "Database name: foo", no
Done


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@397
PS20, Line 397:   public static synchronized ExternalEventsProcessor getOrCreate(
> typically called "instance()" or "getInstance()".
renamed the method and changed the INSTANCE to instance. However, I do think that this method needs to be synchronized to avoid the risk of double instantiation in the future this is called from multiple threads


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@310
PS21, Line 310: if (currentEventId <= lastSyncedEventId_) {
              :         return Collections.emptyList();
              :       }
> nit: single line
Done


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@373
PS21, Line 373:                             .append("Event id : " + event.getEventId() + "\n")
> weird formatting.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 19 Jan 2019 01:54:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 13:

(24 comments)

Firstly thanks for the clean-up. The new class structure makes it much clearer IMO. Publishing a bunch of minor comments. Please bear with my nits.

Also, I had a brief chat with Vihang offline. It looks like the event handling races is still a WIP. My general concern was around the following 3 areas.

- Races with conflicting local events.
- What if Catalog already has the latest HMS state (due to a previous invalidation)
- Failure scenarios if some event processing errors out. (Should we stop and invalidate everything, should be abort the thread etc.)

Vihang mentioned that he is looking into HMS side versioning for objects to detect if the Catalog server already has the latest state and we can skip applying certain events. That probably requires some Hive side changes. Also we discussed about incorporating some of his thought process into class comments.

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@304
PS8, Line 304: 
             :   private void initializeMetastoreEventProcessor() {
             :     long eventPollingInterval = BackendConfig.INSTANCE.getHMSPolli
> done. Did you mean changing the log level to fatal as well?
fatal() IIRC aborts the process. If metastore polling is configured  and we are not able to init here, it looks to me like we should abort (instead of silently swallowing) since that can result in correctness issues later and is unexpected from a user POV. Thoughts?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@292
PS13, Line 292: // start polling for metastore events
              :     if (metastoreEventProcessor_ != null) {
              :       metastoreEventProcessor_.start();
              :     }
Any reason for not moving this to initializeMetastoreEventProcessor()?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@308
PS13, Line 308: debug
nit: Make this info?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1460
PS13, Line 1460:       if (tblToBeRemoved != null) {
nit: invert the logic?

if (tblToBeRemoved == null) return null;
....


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@38
PS13, Line 38:   static abstract class MetastoreEventHandler {
Add doc about the thread-safety expectations here.


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@39
PS13, Line 39: CatalogServiceCatalog catalog,
Any reason for passing this around? Can't we instantiate this once and apply changes to it?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@42
PS13, Line 42:     protected String dbName;
Document what it means to have these set. It looks like the dbName is always set but tblName is optional


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@43
PS13, Line 43:     protected String tblName;
nit: _ for class members


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@47
PS13, Line 47:     protected void validateAndInit(String expectedType, NotificationEvent event) {
method doc.


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@49
PS13, Line 49: event.getTableName()
wouldn't this be null in CREATED_DB/DROP_DB case?

Also, is it worth adding a TRACE log that dumps the event debug data? (Impala allows dynamically switching log levels for debugging).


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@74
PS13, Line 74:   static class CreateTableEventHandler extends MetastoreEventHandler {
method docs for each event handlers?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventHandlerUtils.java@78
PS13, Line 78: CREATE_TABLE
nit: Define these event types as static final globally and reuse them? I think that makes is more readable.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@182
PS8, Line 182: econditions.checkState(pollingFrequencyInSec >= 0);
             :     pollingFrequencyInSec_ = pollingFrequencyInSec;
             :   }
             : 
> The getCurrentNotificationEventId is much cheaper (and hence faster too) th
Fair point.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@347
PS8, Line 347: 
> How do you envision this mechanism? Metastore does not provide any API to f
Nvm, I thought this was HiveClient configuration. Thanks for the correction.


http://gerrit.cloudera.org:8080/#/c/12118/8/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@402
PS8, Line 402: 
> This is not a client side property but metastore server side config. The ge
Gotcha. I thought this was client side configuration.


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@68
PS13, Line 68:  * Events which are not self-events, should be applied on a best-effort basis. All the
             :  * operations which change the state of catalog cache while processing a certain event
             :  * type must be atomic in nature. We rely on taking a write lock on version object in
             :  * CatalogServiceCatalog to make sure that readers are blocked while the metadata update
             :  * operation is being performed. Since the events are generated post-metastore operations,
             :  * such catalog updates do not need to update the state in Hive Metastore
             :  *
             :  * Additionally, in case of CREATE and DROP events, it is important to make sure that past
             :  * event information is not used to modify the future state of the cache. Eg. consider a
             :  * operation sequence of CREATE, DROP and CREATE on a table. In this case, when the first
             :  * two events (CREATE, DROP) are being processed, catalog will already have a table of the
             :  * same name in its cache. In case of such conflicts, event processor should look at the
             :  * creationTimeStamp of the metastore object to make sure that a object from future is not
             :  * modified by object from past.
             :  *
             :  * Following actions are currently taken based on the type of events received from
             :  * metastore.
             :  * <dl>
             :  * <dt>CREATE_TABLE, CREATE_DATABASE</dt>
             :  * <dd>A new table/database is created in Catalog respectively. In case a
             :  * table[database] with the same name already exists in catalog, we compare the createTime
             :  * of the table[database] to resolve the conflict. The newly created table/database are
             :  * Incomplete and should get loaded lazily when needed</dd>
             :  * <dt>DROP_TABLE, DROP_DATABASE</dt>
             :  * <dd>The table/database is dropped from catalog if they are present and if the
             :  * createTime of the object matches with the object present in the event</dd>
             :  * <dt>ALTER_TABLE</dt>
             :  * <dd>In case of alter table event, currently the code issues a invalidate table
             :  * command.
             :  * There is a special case of this event in case of renames, where the old table is
             :  * removed and a new IncompleteTable with the new name is created.</dd>
             :  * <dt>ALTER_DATABASE</dt>
             :  * <dd>In case of alter database events, currently only the case of changing default
             :  * location,owner and change in database description is supported.</dd>
             :  * <dt>ADD_PARTITION, ALTER_PARTITION and DROP_PARTITION</dt>
             :  * <dd>Currently, in case of these events, we issue invalidate on the table. This can be
             :  * optimized by issuing add/refresh/drop the partition at the Table level</dd>
             :  * <dt>All other events are currently ignored</dt>
             :  * </dl>
Move the  content from here to the event handlers impl?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@139
PS13, Line 139: CreateDatabaseEventHandler
Should we make them singletons?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@195
PS13, Line 195:     if (isInitialized_) {
nit: use AtomicBoolean#compareAndSet(). The current impl is theoretically racy.


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@236
PS13, Line 236: if (currentEventId <= lastSyncedEventId_) {
              :         return;
              :       }
nit: single line without braces.


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@251
PS13, Line 251: if (eventHandlers.containsKey(event.getEventType())) {
              :           eventHandlers.get(event.getEventType()).process(catalog_, event);
              :         } else {
              :           eventHandlers.get(DEFAULT_EVENT_HANDLER_KEY).process(catalog_, event);
              :         }
I think you can simplify this with, 

eventHandlers.getOrDefault(event.getEventType(), defaultHandler).process();


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@262
PS13, Line 262:         // Make sure to update lastSyncedEventId_ in case there are errors while
One question around this error handling, is it safe to continue at this point if the subsequent events depend on the failed event?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@283
PS13, Line 283:   /**
nit: newline


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@298
PS13, Line 298:     if (INSTANCE == null) {
how about reversing this. (fewer indents)

if (INSTANCE != null) return INSTANCE;
return INSTANCE = new MetaStore...();


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java:

http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreNotificationException.java@25
PS13, Line 25: class MetastoreNotificationException extends ImpalaException {
nit: The general practice is to explicitly qualify objects with public/private keywords. Looks like it is needed in multiple places in the patch. (AFAIK Impala doesn't follow the package-private semantics)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Jan 2019 00:51:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 27:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/12118/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@448
PS27, Line 448:                 + "invalidated since it does not exist anymore", getFullyQualifiedTblName()));
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/27/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@464
PS27, Line 464:         Pair<Boolean, Boolean> result = catalog_.renameOrAddTableIfNotExists(oldTTableName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/27/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/27/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@758
PS27, Line 758:   private void createDatabaseFromImpala(String dbName, String desc) throws ImpalaException {
line too long (92 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 27
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 01:34:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 20:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1809/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 00:47:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 21:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1812/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 01:47:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1706/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 13
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Jan 2019 00:43:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 9:

(22 comments)

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@294
PS9, Line 294:    * Initializes Metastore event processor object if 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@46
PS9, Line 46:  * <code>org.apache.hadoop.hive.metastore.api.NotificationEvent</code>. Metastore can be configured,
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@47
PS9, Line 47:  * to work with Listeners which are called on various DDL operations like create/alter/drop
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@48
PS9, Line 48:  * operations on database, table, partition etc. Each event has a unique incremental id and the
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@49
PS9, Line 49:  * generated events are be fetched from Metastore to get incremental updates to the metadata stored
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@50
PS9, Line 50:  * in Hive metastore using the the public API <code>get_next_notification</code> These events could
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@51
PS9, Line 51:  * be generated by external Metastore clients like Apache Hive or Apache Spark as well as other
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@54
PS9, Line 54:  * This class is used to poll metastore for new events at a given frequency. By applying such events
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@55
PS9, Line 55:  * on the catalogD we can sync to external metadata operations by taking appropriate actions for
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@56
PS9, Line 56:  * each event type. We keep track of the last synced event id in each polling iteration so the next
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@64
PS9, Line 64:  *   A new table/database is created in Catalog respectively. The newly created table/database are
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@69
PS9, Line 69:  *   In case of alter table event, currently the code issues a invalidate table command. There is a
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@70
PS9, Line 70:  *   special case of this event in case of renames, where the old table is removed and a new
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@73
PS9, Line 73:  *   In case of alter database events, currently only the case of changing default location, owner
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@76
PS9, Line 76:  *   Currently, in case of these events, we issue invalidate on the table. This can be optimized by
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@162
PS9, Line 162:       // fetch the current notification event id. We assume that the polling interval is small
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@163
PS9, Line 163:       // enough that most of these polling operations result in zero new events. In such a case,
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@164
PS9, Line 164:       // fetching current notification event id is much faster (and cheaper on HMS side) instead
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@186
PS9, Line 186:             .format("Unable to process notification event %d due to %s. Ignoring this event...",
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@189
PS9, Line 189:         // Make sure to update lastSyncedEventId_ in case there are errors while processing the
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@190
PS9, Line 190:         // event. Otherwise, the sync thread will be stuck forever at this bad event and keep
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java:

http://gerrit.cloudera.org:8080/#/c/12118/9/fe/src/test/java/org/apache/impala/catalog/MetastoreEventsProcessorTest.java@72
PS9, Line 72:       eventsProcessor = MetastoreEventsProcessor.getOrCreate(catalog, currentNotificationId.getEventId(), 0L);
line too long (110 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 9
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Dec 2018 00:58:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 19:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1806/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 21:42:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 27:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/1913/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 27
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jan 2019 17:50:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 21:

(32 comments)

Very nice improvements: the event classes greatly simplify the logic.

This round focused a bit on error reporting and logging. Many specific comments. As always, these are just questions and suggestions, you should decide which make sense.

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@324
PS21, Line 324:           e);
Will the code that handles this exception write to the log both the message from this exception AND the message from the "cause" exception?

The goal is that, if a failure occurs, we get the detailed error message as well as the "summary" message.

If we log the stack trace, then we're good.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1216
PS21, Line 1216:     long currentEventId = metastoreEventProcessor_.getCurrentEventId();
Should this be here? Or in the event processor itself? If here, why is it needed other than to pass it along to the event processor?

Best not to leak implementation details if possible.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1360
PS21, Line 1360:       Reference<Boolean> dbWasFound, Reference<Boolean> tblWasFound) {
I still think that using the reference is non-standard Java. A simple solution is:

public Pair<Boolean,Boolean> addTableIfNotExists(String dbName, String tblName) ...

Better would be:

enum TableStatus (DB_NOT_FOUND, TBL_NOT_FOUND, OK)

public TableStatus addTableIfNotExists(String dbName, String tblName) ...

Or, suppose we do this, since this method currently returns void:

* Throw an exception if the DB does not exist.
* Return true/false to indicate if the table was created (true) or already existed (false).


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1363
PS21, Line 1363:     Db db = getDb(dbName);
Let's think about this. We return the flags because we want to know if the DB was found or not. We could just as easily do:

DB db = catalog.get(dbName_);
if (db == null) { handle error; }

The reason we don't do the above is we want to prevent race conditions. But... We do the check here outside the lock. So, we have a race condition anyway.

Should either 1) the DB lookup be done by the caller to simplify the code, or 2) be done in the lock to prevent race conditions?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1372
PS21, Line 1372:         tblWasFound.setRef(true);
I wonder, do we really need to know if the table existed or not? Is it even useful? Consider.

* Impala creates a table. It updates its own cache and informs HMS.
* HMS sends a notification. In processing the notification, we determine that the table already exists.

Or

* A table is added in HMS.
* The user immediately kicks of a query in Impala, causing a load of metadata.
* Impala polls for notifications, learns that the table is new, tries to add it, and finds that the table already exists.

In this cases, do we get anything by logging that the table exists? If not, do we need to return the exists-or-not result?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1475
PS21, Line 1475:       Reference<Boolean> tblWasfound, Reference<Boolean> tblMatched) {
Same comments as above.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@130
PS20, Line 130:   public static abstract class MetastoreEvent {
The new event class hierarchy looks very good.

One nit: some events are DB-level events, others are table-level events. This base class includes the table-level stuff and the reader (and code) must know what table stuff is available and when it is not.

So, a further small improvement would be to create another abstract class, MetastoreTableEvent, to hold the common table stuff. The table class extends this one. DB-level events also extend this one.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@138
PS20, Line 138:     protected final Logger LOG = Logger.getLogger(this.getClass());
protected static final ... MetastoreEvent.class);


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@160
PS20, Line 160:       if (!event.getEventType().equalsIgnoreCase("CREATE_DATABASE")
Move this code to the constructor for the suggested MetastoreTableEvent class to avoid redundantly checking the event type.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@182
PS20, Line 182:     }
Would move to the suggested MetastoreTableEvent class.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@199
PS20, Line 199:     private final String tblName_;
These two fields shadow fields of the same name in the base class. This is generally considered undesirable.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@207
PS20, Line 207:       tblName_ = Preconditions.checkNotNull(event.getTableName());
OK to do the checks, but not super necessary since the code they check is part of this file...


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@241
PS20, Line 241:         LOG.info(String.format("Added a table %s.%s after processing event "
Nit: to make it easier to read/parse log records, maybe:

Event %d, (%s): ...

Where the two args are event id and type. That way all the messages follow a common format in the log.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@271
PS20, Line 271:         throw new MetastoreNotificationException("Unable to parse the alter table "
This is a round-about way to do error handling. Generally, you EITHER

1) Use Preconditions to do the check and throw the exception, or
2) Do the check and throw the exception yourself.

Here, would be better to say:

tableBefore_ = alterTableMessage.getTableObjBefore();
if (tableBefore_ == null) {
  throw new ...Exception("Event %d (%s): Invalid message, missing tableObjBefore.")
}

The additional info will be helpful when someone changes the message on HMS without changing Impala and we have to track down the problem based on a long file.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@293
PS20, Line 293:           if (!invalidateCatalogTable(dbName_, tblName_)) {
If this is a method, then no need to pass the fields since the method has visibility to the fields.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@318
PS20, Line 318:           throw new MetastoreNotificationException(String.format("Could not rename "
Let's think about error handling a bit. There are two broad cases:

1. The system went off the rails and got into an unexpected state. Preconditions are great for this.
2. The system system is in a bad state due to environment factors. The user has to clean up the mess.
3. The user provided bad input. The user will fix it and try again.

Given this framework, are the errors here of the 1st or 2nd category?

For example, this one, not being able to rename a table, seems to put the system in a bad state. But, the following error, trying to rename a table not in the cache, seems relatively benign.

Should we log some errors, but continue, while throwing a fatal error for others? Do we need a MetastoreNotificationFailureException vs. a MetastoreNotificationWarningException? Or, just log warnings? If only log warnings, then exceptions are for the "something is terribly wrong, stop the world" kind of error.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@333
PS20, Line 333:         throw new MetastoreNotificationException(e);
Suggestion: have MetastoreNotificationException take this class, or the message class, as a constructor argument. Have the exception class create a full message by adding the message ID and type as suggested for logs. That way, if the exception message appears in a log, we'll have more context about what went wrong:

super(String.format("Event % (%s): %s", id, messageName, e.getMessage), e)


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@383
PS20, Line 383:       if (removedTable == null) {
Nit:

if (removedTable != null) {
  ...
else if (!tblMatched.getRef()) {
 ...
else if (!tblWasFound.getRef()) {
  ...
}

That is, you can reword this to a chain of if/then rather than a nested if.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@389
PS20, Line 389:           LOG.debug(String.format("Table %s from event id %d was not removed since it "
Here, for example, is a nice "you might want to know" kind of message that does not throw an exception.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@440
PS20, Line 440:     public void process() {
I really like how this turned out. Each event is clearly separated. Very easy to follow what's happening. The comments are also easy to follow because they focus on one specific event. Nicely done.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@205
PS20, Line 205:       eventProcessorStatus_ = EventProcessorStatus.ACTIVE;
Should we write something to the log, perhaps at DEBUG or TRACE levels, that capture the major steps?

And, should we write something at, say, INFO level to confirm that this feature is, in fact, started? Will help with "is the feature really turned on" kinds of Support questions.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@215
PS20, Line 215:     statusLock_.readLock().lock();
Seems that statusLock_ is used only within this class. The only read lock is for this function, else we use a write lock.

I wonder, is this over complicating the code? Can't we just use "synchronized" on each method where we need a write lock?

And, since the status itself is a pointer, just returning it without synchronization is good enough: you'll always get a valid pointer. There is no race condition because the caller holds no lock: with or without a lock the reader gets a value at a point in time that may change in the next instant.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@229
PS20, Line 229:     return lastSyncedEventId_;
Here is an example where we're returning a value without locking, even though the value itself is changed in a write lock. This is why the above method also does not need a lock.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@251
PS20, Line 251:       eventProcessorStatus_ = EventProcessorStatus.STOPPED;
Should we check if the processor is in the run state before moving to stopped? Or, at least, before emitting a log message?


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@253
PS20, Line 253:           "Event processing is paused. Last synced event id is %d", lastSyncedEventId_));
Nit: Paused or stopped? Can we move from STOPPED state back to run state (paused) or is STOPPED terminal (stopped)?


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@265
PS20, Line 265:       return metaStoreClient.getHiveClient().getCurrentNotificationEventId().getEventId();
How is this used? How is it synchronized?

if this is simply retrieved by the client, then passed to start(), then shouldn't start() make this call instead?


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@285
PS20, Line 285:       LOG.info(String.format("Metastore event processing restarted. Last synced event id "
Isn't start() called only once? If not, what is the use case for pausing and resuming?

If started the first time, there will be no previous value. Do we want to say "updated from 0 to 100234"? Or, would it be better to say "started at 1000234"?


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@360
PS20, Line 360:       Preconditions.checkState(lastProcessedEvent != null);
The catch-block handles two types of exceptions. It also handles two sources of exceptions. I wonder if the catch-block mixes the two conditions.

The call to getNextMetastoreEvents() could fail due to, say, an RPC error. In this case, there is no lastProcessedEvent. On the other hand, processing the event could fail and there would be a lastProcessedEvent value.

Should the catch block handle both cases? Different catch blocks for the two kinds of errors?


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@371
PS20, Line 371:   private void dumpEventInfoOnErrors(NotificationEvent event) {
The name indicates why the method might be called. But, the method itself does not really care about that. Maybe call it "dumpEventToLog()" or some such.


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@373
PS20, Line 373:                             .append("Event id : " + event.getEventId() + "\n")
Nit: If using a StringBuilder, write each field as a separate .append() call. Otherwise, Java creates a temporary string builder, appends the two strings, converts the result to a string, and then appends that to your string builder.

Or, you may find it easier to use String.format().


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@376
PS20, Line 376:                             .append("Database name : " + event.getDbName() + "\n");
Another nit: colons usually have no leading space: "Database name: foo", not "Database name : foo".


http://gerrit.cloudera.org:8080/#/c/12118/20/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@397
PS20, Line 397:   public static synchronized ExternalEventsProcessor getOrCreate(
typically called "instance()" or "getInstance()".

Also, since the instance is created here, INSTANCE is not a constant, and so should not be capitalized.

Finally, what are the intended semantics? Is start-up multi-threaded? Or, is it, as is typical, single-threaded? If single threaded, no "synchronized" keyword is needed.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 02:35:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 19:

(1 comment)

Thanks for addressing my comments. I think there might be one (probably rare) race that is possible.

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/19/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@269
PS19, Line 269:       lastSyncedEventId_ =
I think fetching the event ID here can lead to a race where we miss a notification, e.g.:

1. CatalogServiceCatalog.reset() is called
2. CatalogServiceCatalog.reset() loads DB and table metadata, which does not include table X.
3. Table X is added to the HMS, e.g. via Hive
4. This function fetches the last event ID
5. The processor starts processing events, but doesn't see the table X create event.

I think the solution is to fetch the last event ID earlier in CatalogServiceCatalog.reset() and pass it into this function.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 19
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 22:22:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,958 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/20
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 20
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for automatic invalidates by polling metastore events

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for automatic invalidates by polling metastore events
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 8
Gerrit-Owner: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 22 Dec 2018 02:53:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,902 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/16
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 16
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................


Patch Set 21:

(14 comments)

Patch code flow and logic lgtm generally. Just had some nits/clarifications. Publishing these while I'm looking at the tests.

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@307
PS21, Line 307: BackendConfig.INSTANCE.getHMSPollingIntervalInSeconds()
eventPollingInterval


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@319
PS21, Line 319:  LOG.error("Unable to fetch the current notification event id from metastore."
              :               + "Metastore event processing will be disabled.",
              :           e);
nit: Multiple places in the code, try to format in fewer lines. Something like

LOG.error("......."
    + ".....", e)


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1206
PS21, Line 1206:     // Even though we get the current notification event id before stopping the event
clarification: reset() is equivalent to invalidating everything and fetching from scratch from HMS. In that case, do we still restart the event processing from where we left off (before reset()) or can we start with the latest event ID from HMS?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@1804
PS21, Line 1804:     Table incompleteTable = IncompleteTable.createUninitializedTable(db, tblName);
move it after L1807?


http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/13/fe/src/main/java/org/apache/impala/catalog/MetastoreEventsProcessor.java@195
PS13, Line 195: 
> curious to understand why do you think this method has a race?
nvm missed the synchronized part.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@155
PS21, Line 155: event %d of type %s on table %s
I think it is cleaner to define a debugString(NotificationEvent event) {} that formats it cleanly and consistently across all invocations. We can use it to dump event state in multiple places.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@160
PS21, Line 160: if (!event.getEventType().equalsIgnoreCase("CREATE_DATABASE")
              :           && !event.getEventType().equalsIgnoreCase("DROP_DATABASE")) {
              :         tblName_ = Preconditions.checkNotNull(event.getTableName());
              :       } else {
              :         tblName_ = null;
              :       }
nit: Reformat to the following? (fewer branches and use enum)

tblName_ = null;
if (eventType_ == MetaStoreEventType.CREATE_TABLE) {
  tblName_ = ...
}


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@181
PS21, Line 181: tblName_
checkNotNull()?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@198
PS21, Line 198:    private final String dbName_;
              :     private final String tblName_;
aren't these set in the parent c'tor?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@347
PS21, Line 347: droppedTable_
isn't this more like tblTobeDropped_ ?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java@434
PS21, Line 434: //
nit: remove, you are already in a comment block.


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:

http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@310
PS21, Line 310: if (currentEventId <= lastSyncedEventId_) {
              :         return Collections.emptyList();
              :       }
nit: single line


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@334
PS21, Line 334:     // need to take a write lock since event status needs to be updated in case of errors
              :     // TODO this can improved by taking a readlock here and adding a error handling
              :     // logic in the catch clause which takes a writelock and then downgrades to the
              :     // readlock again. The added complexity is not necessary at this time but we may
              :     // have to revisit this in the future
This blocks invalidate metadata since both of them try to get a writelock and the scope of the write lock here is too big and we are also doing RPCs under lock. We had nasty issues around such locks in the past. Any chance you could narrow down the scope or fix this?


http://gerrit.cloudera.org:8080/#/c/12118/21/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@373
PS21, Line 373:                             .append("Event id : " + event.getEventId() + "\n")
weird formatting.



-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 21
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 16:30:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7970 : Add support for metastore event based automatic invalidate

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/12118 )

Change subject: IMPALA-7970 : Add support for metastore event based automatic invalidate
......................................................................

IMPALA-7970 : Add support for metastore event based automatic invalidate

This change adds support to CatalogD to poll metastore events to issue
invalidate on tables automatically. It adds basic infrastructure to poll
HMS notifications events at a configurable frequency using a backend
config called hms_event_polling_interval_s flag. Currently, it issues
invalidate at tables when it received alter events on table and
partitions. It also adds tables/databases and removes tables from
catalogD when it receives create_table/create_database and
drop_table/drop_database events. The default value of
hms_event_polling_interval_s is 0 which disables the feature. A
non-zero value in seconds of this configuration can be used to enable
the feature and set the polling frequency.

In order to process each event atomically, this feature relies on
version lock in CatalogServiceCatalog. It adds new methods in
CatalogServiceCatalog which takes a write lock on version so that
readers are blocked until the catalog state is updated based on the
events. In case of processing events, the metastore operation is already
completed and only catalog state needs to be updated. Hence we do not
need to make new metastore calls while processing the events and only
version lock is sufficient to serialize updates to the catalog objects
based on events. This locking protocol is similar to what is done in
case of DDL processing in CatalogOpExecutor except it does not need to
take metastoreDdlLock since no metastore operations are needed during
event processing.

The change also adds a new test class to test the basic functionality
for each of the event type which is supported.

Note that this feature is still a work in progress and additional
improvements will be done in subsequent patches. By default the feature
is turned off.

Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventUtils.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
A fe/src/main/java/org/apache/impala/catalog/events/MetastoreNotificationException.java
A fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
A fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A fe/src/test/java/org/apache/impala/catalog/events/SynchronousHMSEventProcessorForTests.java
M fe/src/test/resources/postgresql-hive-site.xml.template
13 files changed, 1,974 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/12118/22
-- 
To view, visit http://gerrit.cloudera.org:8080/12118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic70b27780560b7ac9b33418d132b36cd0ca4abf7
Gerrit-Change-Number: 12118
Gerrit-PatchSet: 22
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>