You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2021/07/05 14:23:19 UTC
[Impala-ASF-CR] IMPALA-10502: Handle CREATE/DROP events correctly
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17308 )
Change subject: IMPALA-10502: Handle CREATE/DROP events correctly
......................................................................
Patch Set 15:
(25 comments)
This change makes a lot of sense. I mostly left nit comments.
Looks good overall, but I'm planning to take another look in the following days.
http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@12
PS15, Line 12: For example if we have a sequence of
: CREATE_TABLE, DROP_TABLE, CREATE_TABLE ... on the same table, it is
: possible that when the create table event is being processed, the table
: is not present in catalog because it was dropped recently. In such a
: case, events processor does not have enough state information in
: catalogd to determine that this table has been dropped from the
: catalogd and the event should be ignored.
Maybe it would be easier to understand if we had list the events here, e.g.:
1. CREATE_TABLE foo statement -> Catalog creates table foo
2. DROP_TABLE foo statement -> Catalog drops table foo
... a bit after event processor sees the following event:
3. CREATE_TABLE foo event -> foo is not in Catalog, cannot decide whether we should recreate the table or ignore this event.
http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@29
PS15, Line 29: parition
typo: partition
http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@30
PS15, Line 30: the the
duplicate 'the'
http://gerrit.cloudera.org:8080/#/c/17308/15//COMMIT_MSG@35
PS15, Line 35: Added a new test
Could you please the name of the test?
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2911
PS15, Line 2911: public TCatalogObject reloadHdfsPartition(HdfsTable hdfsTable, String partitionName,
nit: add comment please
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@709
PS15, Line 709: -1L
nit: for readability you could write:
/*createEventId=*/-1L
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@899
PS15, Line 899: =
nit: missing space after =
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@268
PS15, Line 268: null, reason);
nit: fits prev line
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java
File fe/src/main/java/org/apache/impala/catalog/TableLoader.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@86
PS15, Line 86: createdEventId
nit: 'createdEventId' is only used at the callsite, so probably refer to 'eventId'.
Same for next line.
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@87
PS15, Line 87: are not created by
: // Impala
not created by the current CatalogD process?
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java
File fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@33
PS15, Line 33: a
typo: an
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@41
PS15, Line 41: SortedMap
optional: I'd assume that the events would come in more or less sorted order, therefore using a linked list might be more efficient.
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/DeleteEventLog.java@57
PS15, Line 57: a
nit: an
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@589
PS15, Line 589: CatalogException e) {
: if (e instanceof TableLoadingException
: || e instanceof DatabaseNotFoundException) {
Could be:
catch (TableLoadingException | DatabaseNotFoundException e)
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@631
PS15, Line 631: }i
no space between '}' and 'is'
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@632
PS15, Line 632: getFullyQualifiedTblName());
getFullyQualifiedTblName() is passed 2 times.
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1104
PS15, Line 1104:
nit: unnecessary whitespace at the end.
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@108
PS15, Line 108: {code}
nit: at other places in this file we are using <code> and </code>
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@268
PS15, Line 268: .getEvents();
nit: fits prev line
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@272
PS15, Line 272: );
probably add 'e' as well to get more information about the error.
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@547
PS15, Line 547: public List<NotificationEvent> getNextMetastoreEvents(final long eventId,
nit: please add comment
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@580
PS15, Line 580: if
is
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@764
PS15, Line 764: added a
nit: adding an?
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@864
PS15, Line 864: such
: // a event.
nit: such event?
http://gerrit.cloudera.org:8080/#/c/17308/15/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2258
PS15, Line 2258: .equals(event.getEventType()) ||
nit: for readability this could go to the prev line, then the remaining parts of the condition could fit a single line
--
To view, visit http://gerrit.cloudera.org:8080/17308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2c5e96b48abac015240f20295b3ec3b1d71f24a
Gerrit-Change-Number: 17308
Gerrit-PatchSet: 15
Gerrit-Owner: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Jul 2021 14:23:19 +0000
Gerrit-HasComments: Yes