You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org> on 2019/04/25 01:38:15 UTC
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13111
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
IMPALA-7973: Add support for fine grained updates at partition level.
This patch adds suport for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
very expensive for large tables. Here, we refresh partitions in case
of add/drop/alter partition events. HMS processes add/drop partitions
in a transaction. We throw MetastoreNotificationNeedsInvalidateException
if any of the partition refreshes fails.
Testing:
Modified pre-existing tests for partition events to insead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 182 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/1
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3006/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 8
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 18:08:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 19:23:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2927/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 02:31:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 212 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/7
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 7
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2902/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 02:40:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/2978/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:23:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 9
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 22:45:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Reviewed-on: http://gerrit.cloudera.org:8080/13111
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 212 insertions(+), 79 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 10
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13111/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1286
PS1, Line 1286: debugLog("Could not refresh partitions of table {} after a drop_partition event"
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 01:39:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 3:
(15 comments)
Thanks for your comments Vihang and Bharath. Please review my comments.
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7973: Add support for fine grained events processing for
> nit: mention about "events" in the title? Otherwise it looks very general .
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13
PS2, Line 13:
> nit: ..affected..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14
PS2, Line 14: refresh affected partitions
: in case of add/d
> Not sure I understand this, can you please clarify?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19
PS2, Line 19: on to
> nit:typo
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085:
> nit: ..it..
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085: s. Returns true if reload of the partition succeeds,
: * false othe
> Instead of referring to specific methods, rephrase it to something like re
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089
PS2, Line 2089: reloadP
> nit:call "reload" to be consistent ?
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102
PS2, Line 1102: per(catalog, metrics, event);
: Preconditions.checkState(eventType_.equals(MetastoreEventType.ADD_PARTITION));
: if (event.getMessage() == null) {
: throw new IllegalStateException(debugString("Event messag
> nit: Not super clear what this is. How are HMS transactional semantics rela
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106:
> Would be good if you can log a useful information about the event like numb
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: {
: AddPartitionMessage addPartitionMessage_ =
: MetastoreEventsProcessor.getMessageFactory()
: .getDeserializer()
: .getAddPartitionMessage(event.getMessage());
: addedPartitions_ =
: Lists.newArrayList(addPartitionMessage_.getPartitionObjs());
: Preconditions.checkState(addedPartitions_.size() > 0);
: //
> We can avoid one unnecessary conversion from Partition -> partSpecMap -> TP
Created static method in base class to do this.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118
PS2, Line 1118:
> nit: include the partition name too
This scenario is only if the table is not present in the catalog, Next load of the table will refresh all the partitions anyway. Do you think logging which partition failed is useful to the user in this case?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125
PS2, Line 1125:
> include the partition name
In success case, it can be a really log list of partitions. (eg:- dynamically loaded partitions)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: */
: private AlterPartitionEvent(CatalogServiceCatalog catalog, Metrics metrics,
: NotificationEvent event) throws MetastoreNotificationException {
: super(catalog, metrics, event);
: Preconditions.checkState(eventType_.equals(MetastoreEventType.ALTER_PARTITION));
: Preconditions.checkNotNull(event.getMessage());
: AlterPartitionMessage alterPartitionMessage =
: MetastoreEventsProcessor.getMessageFactory().getDeserializer()
: .getAlterPartitionMessage(event.getMessage());
:
> same suggestion as above. Also, instead of duplicating this logic, create a
Done.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1266
PS2, Line 1266:
> add a null check as well
Done
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1295
PS2, Line 1295: ex);
> There seem to be a lot of common code between the partition event processor
debugLog and infoLog are already factored out. Each path has a different message. Do you have suggestions to consolidate it? Thanks
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 26 Apr 2019 01:42:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 6:
Resolved merge conflicts with IMPALA-8454.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:31:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2981/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 23:28:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 3:
(5 comments)
Patch looks good to me. some nits below and its good to go from my side.
http://gerrit.cloudera.org:8080/#/c/13111/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/3//COMMIT_MSG@17
PS3, Line 17: affetced
spell-check
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1081
PS3, Line 1081: org.apache.hadoop.hive.common.FileUtils
Can we import this instead of using fully qualified classname?
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1153
PS3, Line 1153: infoLog
no need to have add_partitions in the log message since infoLog provides that information
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1307
PS3, Line 1307: {} partitions dropped from table {} in alter_partition event
This statement doesn't make sense to me. Why would there be dropped partitions in alter_partition event? Also, the log doesn't need to have a event type in it since infoLog method takes care of it.
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1315
PS3, Line 1315: debugLog
no need to say "after a drop_partition event" here since the debugLog prints both the event id and event type
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 17:31:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 4:
(4 comments)
Thanks Vihang, addressed the comments.
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1081
PS3, Line 1081: foLog(debugString("Database %s was not
> Can we import this instead of using fully qualified classname?
Done
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1153
PS3, Line 1153:
> no need to have add_partitions in the log message since infoLog provides th
Done
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1307
PS3, Line 1307: "partition on table {} partition {} failed. Event processin
> This statement doesn't make sense to me. Why would there be dropped partiti
Done
http://gerrit.cloudera.org:8080/#/c/13111/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1315
PS3, Line 1315: d void i
> no need to say "after a drop_partition event" here since the debugLog print
Done
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:22:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
IMPALA-7973: Add support for fine grained updates at partition level.
This patch adds suport for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
very expensive for large tables. Here, we refresh partitions in case
of add/drop/alter partition events. HMS processes add/drop partitions
in a transaction. We throw MetastoreNotificationNeedsInvalidateException
if any of the partition refreshes fails.
Testing:
Modified pre-existing tests for partition events to insead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 183 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/2
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affetced in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 208 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/3
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 6:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2113
PS6, Line 2113: Throws CatalogException if partition reload is unsuccessful. Throws
: * DatabaseNotFoundException if Db doesn't exist
We should use @throws CatalogException and @throws DatabaseNotFoundException javadoc instead.
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1252
PS6, Line 1252: /**
nit: add a new line after L1251
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1255
PS6, Line 1255: getTPartSpecFromHmsPart
I think it's better to not shorten partition as part since it can be confusing.
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1272
PS6, Line 1272: constructPartStringFromTpart
nit: constructPartitionStringFromTPartition
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1288
PS6, Line 1288: List<Partition> addedPartitions_;
can this be private final?
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1463
PS6, Line 1463: private final List<Map<String, String>> droppedPartitions_;
nit: add a new empty line after this variable declaration
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1523
PS6, Line 1523: a
typo: an
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13111/6/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@1573
PS6, Line 1573: Collection<? extends FeFsPartition> partsAfterAdd =
We don't have to do it in this CR, but the method here is getting too big. We should have a separate helper method for each event. It' makes the code easier to read.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 00:54:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115
PS7, Line 2115: @throws
> typo: @throws
Done
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@551
PS7, Line 551: TPart
> TPartiton is the name of the struct, so it's more readable if we don't shor
Done
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@568
PS7, Line 568: P
> nit: this should be an upper case P
Done
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 8
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 17:22:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 7
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 07:12:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115
PS7, Line 2115: @thorws
typo: @throws
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@551
PS7, Line 551: TPart
TPartiton is the name of the struct, so it's more readable if we don't shorten this. TPartition?
http://gerrit.cloudera.org:8080/#/c/13111/7/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@568
PS7, Line 568: p
nit: this should be an upper case P
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 7
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 14:43:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 9
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2901/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 02:25:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
Patch Set 2:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7973: Add support for fine grained updates at partition level.
nit: mention about "events" in the title? Otherwise it looks very general .
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13
PS2, Line 13:
nit: ..affected..
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14
PS2, Line 14: HMS processes add/drop partitions
: in a transaction
Not sure I understand this, can you please clarify?
http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19
PS2, Line 19: insead
nit:typo
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085:
nit: ..it..
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085
PS2, Line 2085: Returns true if reloadPartitition() succeeds, false
: * otherwise.
Instead of referring to specific methods, rephrase it to something like returns true if the reload of the partition is successful, false otherwise....
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089
PS2, Line 2089: refresh
nit:call "reload" to be consistent ?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102
PS2, Line 1102: // HMS adds partitions in a transactional way. We try to refresh all the
: // partitions that were added to HMS. If any partition refresh fails, we throw
: // MetastoreNotificationNeedsInvalidateException exception. We skip refresh of the
: // partitions if the table is not present in the catalog.
nit: Not super clear what this is. How are HMS transactional semantics related to what we are doing here?
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118
PS2, Line 1118: );
nit: include the partition name too
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125
PS2, Line 1125: ;
include the partition name
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1295
PS2, Line 1295: + "event.", getFullyQualifiedTblName());
There seem to be a lot of common code between the partition event processors. Can we consolidate it by having some helpers that factor out the common code and exception handling?
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 04:43:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 5:
Rebased after insert events patch was merged.
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 21:37:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 4: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13111/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@56
PS4, Line 56: ClassUtil
remove if unused
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Apr 2019 18:52:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 200 insertions(+), 44 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/5
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 8
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition level.
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106: addedPartitions_
Would be good if you can log a useful information about the event like number of partitions, isReplace, tablename etc.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: Map<String, String> partSpec = new HashMap<>();
: List<org.apache.hadoop.hive.metastore.api.FieldSchema> fsList =
: msTbl_.getPartitionKeys();
: List<String> partVals = partition.getValues();
: Preconditions.checkNotNull(partVals);
: Preconditions.checkState(fsList.size() == partVals.size());
: for (int i = 0; i < fsList.size(); i++) {
: partSpec.put(fsList.get(i).getName(), partVals.get(i));
: }
We can avoid one unnecessary conversion from Partition -> partSpecMap -> TPartitionKeyValue if we change the signature of the refreshPartitionIfExists to take List<TPartitionKeyValue> here.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: Preconditions.checkNotNull(partitionAfter_);
: Map<String, String> partSpec = new HashMap<>();
: List<org.apache.hadoop.hive.metastore.api.FieldSchema> fsList =
: msTbl_.getPartitionKeys();
: List<String> partVals = partitionAfter_.getValues();
: Preconditions.checkNotNull(partVals);
: Preconditions.checkState(fsList.size() == partVals.size());
: for (int i = 0; i < fsList.size(); i++) {
: partSpec.put(fsList.get(i).getName(), partVals.get(i));
: }
same suggestion as above. Also, instead of duplicating this logic, create a util method in the base class (or a static method)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1266
PS2, Line 1266: droppedPartitions_
add a null check as well
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Apr 2019 02:14:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
4 files changed, 201 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/6
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 212 insertions(+), 79 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/8
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 8
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Anurag Mantripragada (Code Review)" <ge...@cloudera.org>.
Anurag Mantripragada has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
IMPALA-7973: Add support for fine grained events processing for
partition level HMS events.
This patch adds support for fine grained updates for add/drop/alter
partition events.
Currently, partition events invalidate the table. This can be
expensive for large tables. Here, we refresh affected partitions
in case of add/drop/alter partition events. HMS processes add/drop
partitions in a transaction, which means there may be multiple
partitions affected in a single add/drop event. We try to refresh all
these partitions in a loop. If any of the partition refresh fails,
we throw MetastoreNotificationNeedsInvalidateException to mandate a
manual invalidate for event processing to continue.
Testing:
Modified pre-existing tests for partition events to instead test if
partitions are added/dropped/altered when event processing is enabled.
Change-Id: I213401329f3965dd81055197792ccf8a05368af5
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
3 files changed, 205 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/4
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained events processing for partition level HMS events.
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4121/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 9
Gerrit-Owner: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <an...@cloudera.com>
Gerrit-Reviewer: Bharath Krishna <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 17:24:48 +0000
Gerrit-HasComments: No