You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2022/12/19 17:23:23 UTC
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19378
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
3 files changed, 189 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/1
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 9:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429
PS8, Line 429:
: */
: @VisibleForTesting
: public static List<Long> fireReloadEventHelper(MetaStoreClient msClient,
: boolean isRefresh, List<String> partVals, String dbName, String tableName,
: Map<String, String> selfEventParams) throws TException {
: throw new UnsupportedOperationException("Reload event is not supported.");
: }
:
: /**
> These comments are stale now.
Ack
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@541
PS8, Line 541:
> nit: "getFieldsFromReloadEvent" sounds better since it returns the results
Ack
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6417
PS8, Line 6417: MetastoreShim.fireReloadEventHelper(catalog_.getMetaStoreClient(),
> Just realized that the returned event ids are ignored. Does the self-event
Self-events are detected using the catalog service id and catalog version. Event id will not be used at all. Initially, I thought we can use event id but later, so that is why event ids are present here but I'm not using this. Will remove it now.
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py@244
PS8, Line 244: .format(unique_database, test_reload_table))
> To avoid the test become flaky, we need to wait after this line until the e
Ack
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py@246
PS8, Line 246: def check_self_events(query):
> Can we add an optional argument for the partition values (default to None)
There is no partition info in the notification event. we'll have to deserialize the notification message to obtain this value. So I don't think this is possible in python code
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:12:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 7:
(1 comment)
We can rebase this patch on top of https://gerrit.cloudera.org/c/19399/ to test the build.
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG@7
PS2, Line 7: IMPALA:11808: Added
> nit: "IMPALA-11808: Add ..."
Please fix the JIRA id
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 23:14:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 12:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@7
PS10, Line 7: reload even
> Is there a jira about adding this new event in HMS? It could be mentioned i
Ack
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@9
PS10, Line 9: This patch supports a new event called reload event in catalogD.
> Can you add a few more info?
Ack
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@12
PS10, Line 12: invalidat
> typo
Ack
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@13
PS10, Line 13: Hive jira that addes support for this reload event. This
> A bit of info could be added about testing - currently only event creating
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 16:41:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#15).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used to update table/file metadata for a refresh
or invalidate command in other replicas of catalogDs when one
of the replicas issued refresh/invalidate command. HIVE-26838 is
the Hive jira that adds support for this reload event. This
feature is disabled by default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing of reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 336 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/15
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#7).
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 334 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/7
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 6:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/19378/6/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/6/be/src/catalog/catalog-server.cc@127
PS6, Line 127: This configuration is used "
: "to fire a refresh/invalidate table event to the HMS and other event processors "
: "can process this event. The default value is false, so impala will not fire this "
: "event. If enabled, impala will fire this event and other catalogD will process it
It is not clear to me from the description that this config affects both firing and processing of the event
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@442
PS6, Line 442: return Collections.EMPTY_LIST;
I would prefer throwing UnsupportedOperationException
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@447
PS6, Line 447: MetastoreNotificationException
UnsupportedOperationException seem more fitting here.
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2474
PS6, Line 2474: enableProcessingReloadEvent
Isn't it called enableReloadEvents()?
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2476
PS6, Line 2476: will
will not be processed?
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2531
PS6, Line 2531: //we always treat the table as non-transactional so all the files are reloaded
nit: missing space
http://gerrit.cloudera.org:8080/#/c/19378/6/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/6/tests/custom_cluster/test_events_custom_configs.py@262
PS6, Line 262: check_self_events("refresh {}.{} partition(year=2022)"
nit: +1 black line after the nested function may make it easier to read
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 21:19:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 13:
(7 comments)
Some spelling nits
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@14
PS13, Line 14: feature is disabled dy default using a config
Nit: "by"
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128
PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors "
I don't understand this sentence.
Should "and" be "so"?
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429
PS13, Line 429: * CDP Hive-3 only function.
Should we mention CDP in upstream code?
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439
PS13, Line 439: * CDP Hive-3 only function.
Should we mention CDP in upstream code?
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@505
PS13, Line 505: * Fires an reload event to HMS notification log. In Hive-3 the relaod event
Nit: "a reload", "the reload event"
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@246
PS13, Line 246:
Some spaces here that could be deleted?
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@252
PS13, Line 252: # Check if there is an reload event fired after refresh query.
Nit: "a reload event"
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:42:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12062/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Dec 2022 00:27:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 1:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@499
PS1, Line 499: public static List<Long> fireReloadEventHelper(MetaStoreClient msClient, boolean isRefresh,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@500
PS1, Line 500: List<String> partVals, String dbName, String tableName,
line too long (104 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@501
PS1, Line 501: Map<String, String> selfEventParams) throws TException {
line too long (105 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2509
PS1, Line 2509: catalog_.invalidateTable(tbl.getTableName().toThrift(), tblWasRemoved, dbWasAdded);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2530
PS1, Line 2530: + "partition on table {} partition {} failed. Event processing cannot "
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2531
PS1, Line 2531: + "continue. Issue an invalidate metadata command to reset the event "
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2548
PS1, Line 2548: + "cannot continue. Issue an invalidate metadata command to reset "
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6357
PS1, Line 6357: LOG.error(String.format(HMS_RPC_ERROR_FORMAT_STR, "refreshTable") + e.getMessage());
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 17:24:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#4).
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 298 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/4
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@518
PS4, Line 518: FireEventRequestData data = new FireEventRequestData();
> What is the expectation for other HMS users than Impala when receiving this
Yeah, they'll have to ignore this event.
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6401
PS4, Line 6401: MetastoreShim.fireReloadEventHelper(catalog_.getMetaStoreClient(),
> Do I understand correctly that we do not check whether the table has actual
Yeah, we'll always send a refresh event because there is no way of knowing if, in fact, something has really changed for a refresh to take place.
In this case, we can introduce a config to disable this feature by default. And only enable if this feature is really required in a multi-catalogD environment.
Regarding multiple refreshes, I'll create a follow-up Jira to track the optimization patch for this feature.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 20:32:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#9).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 335 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/9
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
3 files changed, 192 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/2
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 16:
> Patch Set 15:
>
> (1 comment)
Clarified the comment
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 23:16:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#13).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invalidate command. HIVE-26838 is the
Hive jira that adds support for this reload event. This
feature is disabled dy default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing of reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 336 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/13
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 16: Verified+1 Code-Review+1
Latest patchset LGTM. Carry forward previous +1 from Quanlong and Csaba and adding mine. Will bump to +2 after Andrew has another look.
Also carry forward previous Verified +1 since the latest changes are not affecting the code functionality.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 23:26:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used to update table/file metadata for a refresh
or invalidate command in other replicas of catalogDs when one
of the replicas issued refresh/invalidate command. HIVE-26838 is
the Hive jira that adds support for this reload event. This
feature is disabled by default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing of reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Reviewed-on: http://gerrit.cloudera.org:8080/19378
Tested-by: Aman Sinha <am...@cloudera.com>
Reviewed-by: Andrew Sherman <as...@cloudera.com>
Reviewed-by: Aman Sinha <am...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 337 insertions(+), 2 deletions(-)
Approvals:
Aman Sinha: Looks good to me, approved; Verified
Andrew Sherman: Looks good to me, but someone else must approve
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 17
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12098/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 20:18:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12112/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 23:37:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8950/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 10:31:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 15:38:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#12).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invalidate command. HIVE-26838 is the
Hive jira that addes support for this reload event. This
feature is disable dy default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 336 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/12
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 16: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Jan 2023 00:29:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 2:
> Patch Set 2:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/12043/ : Initial code review checks failed. See linked job for details on the failure.
The compilation failure is
22:18:20 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project impala-frontend: Compilation failure
22:18:20 [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:[53,50] cannot find symbol
22:18:20 [ERROR] symbol: class ReloadMessage
22:18:20 [ERROR] location: package org.apache.hadoop.hive.metastore.messaging
We need to bump the CDP_BUILD_NUMBER to contain the related Hive changes.
https://github.com/apache/impala/blob/390a932064ba48c44e70e431f2c20e53beddd970/bin/impala-config.sh#L201
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Dec 2022 02:14:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12058/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Dec 2022 07:48:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@518
PS4, Line 518: FireEventRequestData data = new FireEventRequestData();
What is the expectation for other HMS users than Impala when receiving this event? Will they ignore it because they do not know the new member in the union FireEventRequestData? https://github.com/apache/hive/blob/da42736483ce110997f924a746c63b4bc8a8d5b7/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L1480
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6401
PS4, Line 6401: MetastoreShim.fireReloadEventHelper(catalog_.getMetaStoreClient(),
Do I understand correctly that we do not check whether the table has actually changed during REFRESH and always send an event?
I have some concerns about potential performance impact of the change. If N Impala clusters connect to an HMS and all do REFRESH for a table(or INVALIDATE METADATA + actually use the table), then all REFRESH/INVALIDATE METADATA triggers an event and causes other catalogds to also do the reload, leading to each catalogd doing N refresh.
I think that this can be solved for REFRESH by skipping the event generation if actually nothing has changed in the table. If there are changes (e.g. new files were written to an external table without sending the relevant events to HMS) then it useful to notify other Impalas to refresh their metadata snapshots, but otherwise the REFRESH was NOOP, it doesn't seem meaningful to send an event.
INVALIDATE METADATA seems like a harder case - we don't know whether there were any changes while unnecessarily invalidating + reloading the table on other catalogds can be very expensive. I think that it would be actually the best to not propagate INVALIDATE METADATA to other cluster, at least by default. A query option could be added to set whether to send the event, so someone could initiate a "global" INVALIDATE METADATA, but not do it by default.
http://gerrit.cloudera.org:8080/#/c/19378/4/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/4/tests/custom_cluster/test_events_custom_configs.py@255
PS4, Line 255: .
nit: +4 indentation
http://gerrit.cloudera.org:8080/#/c/19378/4/tests/custom_cluster/test_events_custom_configs.py@258
PS4, Line 258: self.client.execute("drop table {}.{}".format(unique_database, test_reload_table))
there is no need to drop the table, unique_database will be cleaned up by test framework
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Jan 2023 16:01:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/19378/5/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/5/be/src/catalog/catalog-server.cc@127
PS5, Line 127: enable_processing_reload_event
The name sounds like only applying on the processing side and not the triggering side. I think "enable_reload_events" is better since there is a similar flag of enable_insert_events.
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@446
PS5, Line 446: FireEventRequestData data = new FireEventRequestData();
: data.setRefreshEvent(isRefresh);
: FireEventRequest rqst = new FireEventRequest(true, data);
We can't use these if they don't exist in Apache Hive3. We can make this method a no-op and return an empty list. Need to update the comment for this as well.
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2444
PS2, Line 2444: Preconditions.checkArgument(MetastoreEventType.RELOAD.equals(getEventType()));
: Preconditions.checkNotNull(event.getMessage());
: ReloadMessage reloadMessage =
: MetastoreEventsProcessor.getMessageDeserializer()
: .getReloadMessage(event.getMessage());
: try {
: msTbl_ = Preconditions.checkNotNull(reloadMe
> We might need to move this into MetastoreShim if Apache Hive3 doesn't have
This still needs to be addressed.
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6410
PS5, Line 6410: LOG.warn(String.format("Couldn't obtain a version lock for the table: %s",
nit: also mention that reload events won't be fired.
http://gerrit.cloudera.org:8080/#/c/19378/5/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/5/tests/custom_cluster/test_events_custom_configs.py@254
PS5, Line 254: .format(unique_database, test_reload_table))
Can we also verify the fired reload event? We have an example here:
https://github.com/apache/impala/blob/11068d9aebd83be59773f2b1eaa5682053d4428b/tests/metadata/test_event_processing.py#L406-L412
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 03:43:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12103/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 00:23:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@235
PS3, Line 235: e
flake8: F841 local variable 'event_id_before' is assigned to but never used
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@238
PS3, Line 238: ,
flake8: E501 line too long (111 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@240
PS3, Line 240: a
flake8: E501 line too long (116 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@242
PS3, Line 242: a
flake8: E501 line too long (116 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@243
PS3, Line 243: d
flake8: E306 expected 1 blank line before a nested definition, found 0
http://gerrit.cloudera.org:8080/#/c/19378/3/tests/custom_cluster/test_events_custom_configs.py@251
PS3, Line 251: r
flake8: E501 line too long (104 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Dec 2022 07:38:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#6).
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 338 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/6
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12106/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:33:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128
PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors "
> OK still confused.
"to fire a refresh/invalidate table event to the HMS and any other event processors (such as other Impala catalogds) so they can process this event."
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 22:25:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 13:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@10
PS13, Line 10: update
nit: to update ..
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@11
PS13, Line 11: one the
nit: one of the ..
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:41:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@499
PS2, Line 499: public static List<Long> fireReloadEventHelper(MetaStoreClient msClient,
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 22:07:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6357
PS1, Line 6357: resp.getResult().addToUpdated_catalog_objects(addedDb.toTCatalogObject());
> line too long (92 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Dec 2022 07:37:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12091/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 01:39:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 6:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/19378/6/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/6/be/src/catalog/catalog-server.cc@127
PS6, Line 127: This configuration is used "
: "to fire a refresh/invalidate table event to the HMS and other event processors "
: "can process this event. The default value is false, so impala will not fire this "
: "event. If enabled, impala will fire this event and other catalogD will process it
> It is not clear to me from the description that this config affects both fi
The config only affects the firing of the event. Processing will not be affected by this config. Added more details to this config.
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@442
PS6, Line 442: return Collections.EMPTY_LIST;
> I would prefer throwing UnsupportedOperationException
Ack
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@447
PS6, Line 447: MetastoreNotificationException
> UnsupportedOperationException seem more fitting here.
Ack
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2474
PS6, Line 2474: enableProcessingReloadEvent
> Isn't it called enableReloadEvents()?
Will remove this if clause as we don't want to restrict the processing of reload event.
http://gerrit.cloudera.org:8080/#/c/19378/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2476
PS6, Line 2476: will
> will not be processed?
same as above comment
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 21:58:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12099/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 22:09:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12042/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 17:34:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 2:
(20 comments)
The solution looks good to me. I'm going to take a deeper look.
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG@7
PS2, Line 7: IMPALA:11808: Added
nit: "IMPALA-11808: Add ..."
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@499
PS2, Line 499: fireReloadEventHelper
We need the same method in another MetastoreShim:
https://github.com/apache/impala/blob/master/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
BTW, could you add method comments like other methods?
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2435
PS2, Line 2435: private boolean isRefresh_;
Coud you comment that if isRefresh_ is false, it's an invalidate event?
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2441
PS2, Line 2441:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2444
PS2, Line 2444: ReloadMessage reloadMessage =
: MetastoreEventsProcessor.getMessageDeserializer()
: .getReloadMessage(event.getMessage());
: try {
: msTbl_ = Preconditions.checkNotNull(reloadMessage.getTableObj());
: reloadPartition_ = reloadMessage.getPtnObj();
: isRefresh_ = reloadMessage.isRefreshEvent();
We might need to move this into MetastoreShim if Apache Hive3 doesn't have these methods.
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2462
PS2, Line 2462:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2464
PS2, Line 2464:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2468
PS2, Line 2468:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2485
PS2, Line 2485: else {
nit: to keep this method clean, can we extract codes in the else-branch into a method?
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2493
PS2, Line 2493:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2501
PS2, Line 2501: {}
This is always -1.
nit: should be 4 spaces indent at this line
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2531
PS2, Line 2531:
nit: need to fix the indention
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2545
PS2, Line 2545: false
Could you add a comment that we always treat the table as non-transactional so all files are reloaded?
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2548
PS2, Line 2548: debugString("Refresh table {} failed. Event processing "
:
nit: need to fix the indention
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6275
PS2, Line 6275: // Quick check to see if the table exists in the catalog without triggering
: // a table load.
nit: Please move this comment back since it's more about the null check.
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6337
PS2, Line 6337: // fire event for reload
Can we extract the following code into a method? The execResetMetadata() method is too long now. I think in the future, we should also refactor execResetMetadata() to clean it up..
So I think these code should be moved into MetastoreShim since we can't run them on Apache Hive3.
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6341
PS2, Line 6341:
nit: 4 spaces indent
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6354
PS2, Line 6354: tryWriteLock(tbl);
The default timeout for this is 2 hours. It might not worth waiting so long for tracking self events. We can use the underlying method to wait for a shorter time, e.g. 10mins (or make this configurable so we have a way to skip it):
if (req.isIs_refresh() && catalog_.tryLock(tbl, true, 600000)) {
catalog_.addVersionsForInflightEvents(false, tbl, newCatalogVersion);
} else {
LOG.warn(...);
}
BTW, we don't need to add the inflight version for invalidate ops. In such case, 'tbl' is invalidated and no longer valid.
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6356
PS2, Line 6356:
nit: remove space
http://gerrit.cloudera.org:8080/#/c/19378/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6358
PS2, Line 6358: refreshTable
nit: Can we use a more specifit name, e.g. fireReloadEvent ?
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Dec 2022 12:35:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#16).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used to update table/file metadata for a refresh
or invalidate command in other replicas of catalogDs when one
of the replicas issued refresh/invalidate command. HIVE-26838 is
the Hive jira that adds support for this reload event. This
feature is disabled by default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing of reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 337 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/16
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Andrew Sherman, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#14).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used to update table/file metadata for a refresh
or invalidate command in other replicas of catalogDs when one
of the replicas issued refresh/invalidate command. HIVE-26838 is
the Hive jira that adds support for this reload event. This
feature is disabled dy default using a config
enable_reload_events. To use this feature set this config to
true and impala will be able to fire reload events. The
processing of reload events in the event processor is always
enabled. There is an end-to-end test added for this feature
which currently checks firing/creation of the reload event and
self-event check in the event processor. TODO: end-to-end test
should also test this reload event in the event processor.
There is also a follow up jira IMPALA-11822 to track the
optimization patch for this feature.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 336 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/14
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 14:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@10
PS13, Line 10: to upd
> nit: to update ..
Ack
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@11
PS13, Line 11: en one
> nit: one of the ..
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:44:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 13:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@13
PS12, Line 13: adds
> nit: adds
Ack
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@14
PS12, Line 14: disable
> nit: disabled
Ack
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@17
PS12, Line 17: of rel
> nit: 'of' reload events..
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:33:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 12:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@13
PS12, Line 13: addes
nit: adds
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@14
PS12, Line 14: disable
nit: disabled
http://gerrit.cloudera.org:8080/#/c/19378/12//COMMIT_MSG@17
PS12, Line 17: reload
nit: 'of' reload events..
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:16:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 16: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Jan 2023 00:38:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19378/5/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/5/be/src/catalog/catalog-server.cc@127
PS5, Line 127: enable_reload_events, false, "
> The name sounds like only applying on the processing side and not the trigg
Ack
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@446
PS5, Line 446: throws MetastoreNotificationException{
: throw new MetastoreNotificationException("Reload event is not supported.");;
: }
> We can't use these if they don't exist in Apache Hive3. We can make this me
Done
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6410
PS5, Line 6410: LOG.warn(String.format("Couldn't obtain a version lock for the table: %s. " +
> nit: also mention that reload events won't be fired.
Added more log saying that self-event will not be detected.
http://gerrit.cloudera.org:8080/#/c/19378/5/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/5/tests/custom_cluster/test_events_custom_configs.py@254
PS5, Line 254: last_event = events[0]
> Can we also verify the fired reload event? We have an example here:
Done
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 20:13:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#8).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 334 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/8
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-11808: Add s
> Please fix the JIRA id
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 00:04:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429
PS8, Line 429: So empty list is returned
: *
: * @param msClient Metastore client,
: * @param isRefresh if this flag is set to true then it is a refresh query, else it
: * is an invalidate metadata query.
: * @param partVals The partition list corresponding to
: * the table, used by Apache Hive 3
: * @param dbName
: * @param tableName
: * @return a list of eventIds for the reload events
These comments are stale now.
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@541
PS8, Line 541: updateFieldsForReloadEvent
nit: "getFieldsFromReloadEvent" sounds better since it returns the results but not updates anything.
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6417
PS8, Line 6417: MetastoreShim.fireReloadEventHelper(catalog_.getMetaStoreClient(),
Just realized that the returned event ids are ignored. Does the self-event check of reload events depend on the event id, or depend on catalog version? I see we are passing event ids to the SelfEventContext
https://gerrit.cloudera.org/c/19378/8/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#2466
I'm not that clear how the self-events are skipped in the test. I might miss something..
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py@246
PS8, Line 246: def check_self_events(query):
Can we add an optional argument for the partition values (default to None) and check it?
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 01:13:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#10).
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
IMPALA-11808: Add support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 336 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/10
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 10: Code-Review+1
(2 comments)
LGTM. Thanks for adding this feature!
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19378/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@6417
PS8, Line 6417: MetastoreShim.fireReloadEventHelper(catalog_.getMetaStoreClient(),
> Self-events are detected using the catalog service id and catalog version.
Ack
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py@246
PS8, Line 246:
> There is no partition info in the notification event. we'll have to deseria
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:50:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 10: Code-Review+1
(5 comments)
The code looks good to me, some comments for the commit message
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@7
PS10, Line 7: reload even
Is there a jira about adding this new event in HMS? It could be mentioned in the commit message
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@9
PS10, Line 9: This patch supports a new event called reload event in catalogD.
Can you add a few more info?
- both sending and processing of the event is implemented in this patch
- the flag could be mentioned and that it is disabled by default
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@12
PS10, Line 12: invlidate
typo
http://gerrit.cloudera.org:8080/#/c/19378/10//COMMIT_MSG@13
PS10, Line 13:
A bit of info could be added about testing - currently only event creating and self event checking is implemented. It would be great to have tests for the event processing too - a TODO could be added about this.
http://gerrit.cloudera.org:8080/#/c/19378/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@549
PS10, Line 549: throws MetastoreNotificationException{
nit: extra space
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 10:11:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 294 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/3
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 13:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128
PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors "
> Let's say there are two replicas of impala talking to HMS, One impala servi
OK still confused.
Is it
"to fire a refresh/invalidate table event to the HMS which in turn generates events so that other event processors (such as other Impala catalogds) can process this event."
or
"to fire a refresh/invalidate table event to the HMS and any other event processors (such as other Impala catalogds) so they can process this event."
or something else :-)
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429
PS13, Line 429: * CDP Hive-3 only function.
> Just following some other APIs comments in the same class. Refer L#552 L#56
OK
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439
PS13, Line 439: * CDP Hive-3 only function.
> Just following some other APIs comments in the same class. Refer L#552 L#56
OK
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 22:13:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12111/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 22:20:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 15:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19378/13//COMMIT_MSG@14
PS13, Line 14: feature is disabled by default using a config
> Nit: "by"
Ack
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/19378/13/be/src/catalog/catalog-server.cc@128
PS13, Line 128: "to fire a refresh/invalidate table event to the HMS and other event processors "
> I don't understand this sentence.
Let's say there are two replicas of impala talking to HMS, One impala service would fire the event and the other impala service (event processor) would process this event. Does it make sense?
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@429
PS13, Line 429: * CDP Hive-3 only function.
> Should we mention CDP in upstream code?
Just following some other APIs comments in the same class. Refer L#552 L#560 L#568 L#577
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java@439
PS13, Line 439: * CDP Hive-3 only function.
> Should we mention CDP in upstream code?
Just following some other APIs comments in the same class. Refer L#552 L#560 L#568 L#577
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/19378/13/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@505
PS13, Line 505: * Fires a reload event to HMS notification log. In Hive-3 the relaod event
> Nit: "a reload", "the reload event"
Ack
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@246
PS13, Line 246:
> Some spaces here that could be deleted?
Ack
http://gerrit.cloudera.org:8080/#/c/19378/13/tests/custom_cluster/test_events_custom_configs.py@252
PS13, Line 252: # Check if there is a reload event fired after refresh query.
> Nit: "a reload event"
Ack
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 22:00:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/12043/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Dec 2022 22:18:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/8/tests/custom_cluster/test_events_custom_configs.py@244
PS8, Line 244: .format(unique_database, test_reload_table))
To avoid the test become flaky, we need to wait after this line until the events of previous statements are proccessed. I.e. need a line of
EventProcessorUtils.wait_for_event_processing(self)
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 01:16:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19378/10/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/10/tests/custom_cluster/test_events_custom_configs.py@246
PS10, Line 246:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/19378/10/tests/custom_cluster/test_events_custom_configs.py@246
PS10, Line 246:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:14:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19378/9/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/19378/9/tests/custom_cluster/test_events_custom_configs.py@246
PS9, Line 246: d
flake8: E306 expected 1 blank line before a nested definition, found 0
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:11:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/12105/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 03:30:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11808: Add support for reload event in catalogD
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/19378 )
Change subject: IMPALA-11808: Add support for reload event in catalogD
......................................................................
Patch Set 11: Code-Review+1
Started the tests without giving +2 to see if there is any error.
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 10:32:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA:11808: Added support for reload event in catalogD
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Abhishek Rawat, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19378
to look at the new patch set (#5).
Change subject: IMPALA:11808: Added support for reload event in catalogD
......................................................................
IMPALA:11808: Added support for reload event in catalogD
This patch supports a new event called reload event in catalogD.
This event is used update table/file metadata for a refresh or
invalidate command in other replicas of catalogDs when one the
replicas issued refresh/invlidate command.
Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M tests/custom_cluster/test_events_custom_configs.py
9 files changed, 322 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/19378/5
--
To view, visit http://gerrit.cloudera.org:8080/19378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic62d58837d356dc2113f3c0904228ac9de484136
Gerrit-Change-Number: 19378
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>