You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2023/06/08 00:00:37 UTC
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20022
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
5 files changed, 128 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/1
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#5).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
7 files changed, 154 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/5
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 14:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@323
PS13, Line 323:
> flake8: E306 expected 1 blank line before a nested definition, found 0
Ack
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@348
PS13, Line 348:
> flake8: E501 line too long (94 > 90 characters)
Ack
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@353
PS13, Line 353:
> flake8: E501 line too long (93 > 90 characters)
Ack
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@360
PS13, Line 360:
> flake8: E501 line too long (94 > 90 characters)
Ack
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@372
PS13, Line 372:
> flake8: E501 line too long (93 > 90 characters)
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Aug 2023 22:53:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 12:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3229
PS12, Line 3229: Long
nit: long
BTW, why we don't verify 'lastSyncEventIdBefore' now?
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3239
PS12, Line 3239: assertTrue
nit: use assertEquals()
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@345
PS11, Line 345: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.client.execute(
: "refresh {}.{}".format(unique_database, test_old_table_1))
> The test is running fine. I ran the test 10 times consecutively and it pass
Ack
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@374
PS12, Line 374: "create transactional table {}.{} (i int) partitioned by (year int)"
Can we also test for non-partitioned tables? We can extract a common method to test on the 4 cases of transactional or not and partitioned or not. The method creates the table, then alters and inserts it in Hive and refreshes it in Impala, verify no reloads happen.
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@384
PS12, Line 384: assert tbl_events_skipped_after > tbl_events_skipped_before
Let's verify no reloads happen by comparing the old and new metrics of "tables-refreshed" and "partitions-refreshed".
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Aug 2023 10:54:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20022/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/2/tests/custom_cluster/test_events_custom_configs.py@335
PS2, Line 335: "
flake8: F522 '...'.format(...) has unused named argument(s): db, tbl
http://gerrit.cloudera.org:8080/#/c/20022/2/tests/custom_cluster/test_events_custom_configs.py@335
PS2, Line 335: "
flake8: F524 '...'.format(...) is missing argument(s) for placeholder(s): 0, 1
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Jul 2023 23:38:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13727/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Aug 2023 01:23:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG@14
PS1, Line 14: eventTime
Do we really compare the eventTime?
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG@19
PS1, Line 19: By comparing the lastRefreshEventId to current event id in the event
How do we deal with batch events? For partition level INSERT/ALTER events, we now process them in batch if they are consecutive and belong to the same table. It seems we need to skip some events when adding them in a batch.
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1915
PS1, Line 1915: long latestEventId = getLatestEventId(client);
please log this so we can reason about why an older event is skipped.
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2883
PS1, Line 2883: long latestEventId = getLatestEventId(client);
Please log this so we can reason about why an older event is skipped.
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/Table.java@1048
PS1, Line 1048: debug
Let's bump this to INFO level to ease debug.
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1540
PS1, Line 1540:
nit: remove blank line
http://gerrit.cloudera.org:8080/#/c/20022/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/1/tests/custom_cluster/test_events_custom_configs.py@321
PS1, Line 321: """alter table {db}.{tbl} set tblproperties ('foo'='bar');"""
ALTER event generated by this might already be ignored by the change of IMPALA-11534. Can we use INSERT?
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 Jun 2023 07:53:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Reviewed-on: http://gerrit.cloudera.org:8080/20022
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 227 insertions(+), 9 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 17
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13602/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Jul 2023 23:51:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 16: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Aug 2023 00:41:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13785/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 20 Aug 2023 19:50:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1491
PS6, Line 1491: */
: @Override
: public void process() throws MetastoreNotificationException, CatalogException {
: if (isRename_) {
: processRename();
: return;
: }
:
: if (isOlderEvent(null)) {
:
> I am trying to understand the RENAME case. I think that it would be clearer
Ack
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2341
PS6, Line 2341: rivialAlterP
> I am trying to understand what will happen in the batched case.
No, I'm iterating over each of the partitions and then checking if the partition's eventId is greater than the current eventId.
Sure, I can add the above-mentioned test.
http://gerrit.cloudera.org:8080/#/c/20022/6/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/6/tests/custom_cluster/test_events_custom_configs.py@315
PS6, Line 315: catalogd_args
> Is there a reason why enable_sync_to_latest_event_on_ddls is true in the Ja
Some of the events depend on enable_sync_to_latest_event_on_ddls to be enabled. In this particular test, we don't actually need it.
Once, the performance and stability of catalogD features are completed, we plan to turn on these configs by default.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Aug 2023 19:19:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#7).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 201 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/7
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#8).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 201 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/8
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#15).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 227 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/15
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Aug 2023 20:37:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 4:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1285
PS3, Line 1285: refreshUpdatedPartitions, partitionToEventId, debugAction);
> This and the comment at L1298 confuse me since no codes after them are actu
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1923
PS3, Line 1923: for (HdfsPartition.Builder p : partBuilders) {
> I think we should get this before loadFileMetadataForPartitions() as well.
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1930
PS3, Line 1930: p.setLastRefreshEventId(latestEventId);
> We'd better check and only update it if 'latestEventId' is not -1 or lower
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1932
PS3, Line 1932: }
> Let's add the table name in the log.
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2947
PS3, Line 2947:
> This seems a static method and can be moved to MetastoreEventsProcessor. Ac
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1053
PS3, Line 1053: String messageStr = partitionEventObj == null ? "Skipping the event since the" +
> We'd better add more info about why an event is skipped or not skipped. The
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1529
PS3, Line 1529: } else {
> This happens after we reload the table metadata which is unsafe if there ar
I'll fetch the current event id before l#1522 and set it after HDFS#load(). I fear that we might lose these values if we set these values before loading the metadata.
Why is it any different for non-hdfs tables? we treat all the tables as same whenever we are deciding, whether to skip the event or not right?
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1537
PS3, Line 1537: }
> It seems too strict to fail the DDL just for failures in getting the latest
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1615
PS3, Line 1615: tbl.load(true, msClient.getHiveClient(), msTbl, "ALTER VIEW");
> The same as above, I think this should run before the load() and exceptions
Ack
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3219
PS3, Line 3219: setEnableSyncToLatestEventOnDdls(true)
> Is this required to turn on this optimization?
Unfortunately yes. I'm reusing currentEventId fetched from this flag in reload table logic here:
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2579
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2619
http://gerrit.cloudera.org:8080/#/c/20022/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/3/tests/custom_cluster/test_events_custom_configs.py@316
PS3, Line 316: older_events(self, unique_d
> Do we need this flag?
Actually no. I'll take out this flag.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Aug 2023 01:47:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#4).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
7 files changed, 146 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/4
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13687/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Aug 2023 02:15:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 11:
(8 comments)
The solution looks good to me. Mainly focus on reviewing the tests.
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2939
PS11, Line 2939: partitions
nit: let's also log the number of reloaded partitions, e.g. partBuilderToPartitions.size()
Setting the latest refresh event id to 12345 for the reloaded 100 partitions
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3224
PS11, Line 3224: HdfsTable testTbl = (HdfsTable)catalog_.getOrLoadTable(TEST_DB_NAME, testTblName,
: "test", null);
: eventsProcessor_.processEvents();
These are done in testInsertEvents(). I think we don't need them.
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS11, Line 3233: Long
nit: long
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS11, Line 3233: lastSyncEvenIdBefore
typo: "Even" -> "Event"
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3235
PS11, Line 3235: assertTrue
nit: use assertEquals() to have better error message
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3246
PS11, Line 3246: }
Can we test that REFRESH will also update LastRefreshEventId? E.g. do some Hive operations and run a REFRESH in Impala. Then process the events and verify they are all skipped.
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@324
PS11, Line 324: "create table {}.{} (i int) partitioned by (year int) "
Can we also test transactional tables?
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@345
PS11, Line 345: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.client.execute(
: "refresh {}.{}".format(unique_database, test_old_table_1))
Could you double check these can finish in 5s? If not, some events might be processed which makes this test flaky. We might need a longer hms_event_polling_interval_s
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Aug 2023 07:46:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#12).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 232 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/12
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 16:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9650/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Aug 2023 00:41:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13608/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Jul 2023 06:30:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13817/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Aug 2023 05:21:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 11:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py@348
PS7, Line 348: f.run_stmt_
> Can you also add a test with dynamic partitioning?
Done
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py@357
PS7, Line 357: ocessorUtils.get_num_skip
> Can't we use fixed numbers in the test, e.g. assert tbl_events_skipped_
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 20 Aug 2023 19:24:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#13).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/util/event_processor_utils.py
12 files changed, 234 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/13
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 4:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1527
PS4, Line 1527: reloadFileMetadata, reloadTableSchema, false, partitionsToUpdate,
: debugAction, partitionToEventId, reason);
Shouldn't we consider reloadFileMetadata/reloadTableSchema/partitionsToUpdate?
My concern is that we may update LastRefreshEventId() even for a partial reload of the table and ignore events that would trigger a full reload.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1532
PS4, Line 1532: // Update the lastRefreshEventId at the table level if it is unpartitioned table
: // if it is partitioned table, partitions are updated in HdfsTable#load() method
This also applies to other table types, e.g. Iceberg/Kudu/HBase, right?
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3222
PS4, Line 3222: createTable(testTblName, true);
Both this and the Python test used a partitioned table. As this feature is implemented differently for non-partitioned tables, it would be nice to test them too.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3231
PS4, Line 3231: alterTableAddParameter(testTblName, "somekey", "someval");
Why is calling alterTable from Impala relevant here?
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS4, Line 3233: testTbl = (HdfsTable)catalog_.getOrLoadTable(TEST_DB_NAME, testTblName,
: "test", null);
: assertTrue(testTbl.getLastRefreshEventId()
Shouldn't we load a table and call getLastRefreshEventId before eventsProcessor_.processEvents();?
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Aug 2023 13:55:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#6).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
7 files changed, 157 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/6
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#9).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 204 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/9
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#14).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 238 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/14
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13870/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Aug 2023 16:27:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 1:
Hello everyone, Can you please review this patch?
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Jun 2023 16:51:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#11).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 213 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/11
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1527
PS4, Line 1527: reloadFileMetadata, reloadTableSchema, false, partitionsToUpdate,
: debugAction, partitionToEventId, reason);
> Shouldn't we consider reloadFileMetadata/reloadTableSchema/partitionsToUpda
We always reloadFileMetadat, reloadTableSchema in catalogOpExecutor whenever there is a schema change or file metadata change. If you see the methods calling loadTableMetadata(), we always set them to true whenever they are changed.
Regarding PartitionsToUpdate, we set the lastRefreshEventId on updated partitions only (see HdfsTable.java table L#1930) and never set it at the table level.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1532
PS4, Line 1532: // Update the lastRefreshEventId at the table level if it is unpartitioned table
: // if it is partitioned table, partitions are updated in HdfsTable#load() method
> This also applies to other table types, e.g. Iceberg/Kudu/HBase, right?
For unpartitioned tables, Yes.
For partitioned tables in Iceberg, kudu, there is no way of setting lastRefreshEventID at partition level.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3222
PS4, Line 3222: createTable(testTblName, true);
> Both this and the Python test used a partitioned table. As this feature is
Ack
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3231
PS4, Line 3231: alterTableAddParameter(testTblName, "somekey", "someval");
> Why is calling alterTable from Impala relevant here?
Ideally, not really. Just want to emphasize that lastrefreshEventId would skip this event.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS4, Line 3233: "test", null);
: eventsProcessor_.processEvents();
: assertTrue(testTbl.getLastRefreshEventId()
> Shouldn't we load a table and call getLastRefreshEventId before eventsProce
Ack. we would want to compare currentEventId of event processor to be equal to the lastrefreshEventId after the refresh event. Made the necessary change.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 03:09:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1527
PS4, Line 1527: reloadFileMetadata, reloadTableSchema, false, partitionsToUpdate,
: debugAction, partitionToEventId, reason);
> We always reloadFileMetadat, reloadTableSchema in catalogOpExecutor wheneve
I am still trying to understand what will happen here. For example when calling from alterCommentOnColumn:
https://github.com/apache/impala/blob/8329e6f2e39463d01b549ae84518d131b848adff/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L7315
This sets reloadFileMetadata to false, which is logical for a DDL that changes only column metadata. But what happens if there is an unprocessed insert event? Won't we set lastRefreshEventId to the latest event id (so after the insert event) and skip the insert event as a result?
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS4, Line 3233: "test", null);
: eventsProcessor_.processEvents();
: assertTrue(testTbl.getLastRefreshEventId()
> Ack. we would want to compare currentEventId of event processor to be equal
Wouldn't it be better to save the value of testTbl.getLastRefreshEventId() before eventsProcessor_.processEvents(); and compare with that afterwards? My concern is that if we do not skip the event, then the event processing will trigger a reload that also sets lastRefreshEventId_ so we do not verify that getOrLoadTable() actually sets lastRefreshEventId_
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 09:30:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 14:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py@337
PS14, Line 337: EventProcessorUtils.wait_for_event_processing(self)
Can we also verify no table/partitions are refreshed here? We can extract the following pattern into a method to simplify the code:
def verify_skipping_hive_stmt_events(stmt):
self.run_stmt_in_hive(stmt)
self.client.execute("refresh ...")
get event metrics
EventProcessorUtils.wait_for_event_processing(self)
get event metrics and compare them
'stmt' can be multiple statments like 3 INSERTs.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Aug 2023 02:29:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1285
PS3, Line 1285: // update refresh event id at partition level
This and the comment at L1298 confuse me since no codes after them are actually updating the refresh event id. Can we remove them?
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1923
PS3, Line 1923: long latestEventId = getLatestEventId(client);
I think we should get this before loadFileMetadataForPartitions() as well.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1930
PS3, Line 1930: p.setLastRefreshEventId(latestEventId);
We'd better check and only update it if 'latestEventId' is not -1 or lower than the current tracked refresh event id.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1932
PS3, Line 1932: LOG.info("Setting the latest refresh event id to {} for the loaded partitions",
Let's add the table name in the log.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2947
PS3, Line 2947: public long getLatestEventId(IMetaStoreClient client) {
This seems a static method and can be moved to MetastoreEventsProcessor. Actually, there is a similar method: MetastoreEventsProcessor.getCurrentEventId(). We can add a new one like getCurrentEventIdNoThrow() for this.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1053
PS3, Line 1053: infoLog("Incremented events skipped counter to {}",
We'd better add more info about why an event is skipped or not skipped. They would be useful in debugging issues when we lose track of the event ids, e.g. IMPALA-12256
We already have partition level logs at https://github.com/apache/impala/blob/fed883379383b368bbce4f3b36b95e01a7378578/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L3903
So we just need to add the table level refresh event id.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1529
PS3, Line 1529: eventId = getCurrentEventId(msClient);
This happens after we reload the table metadata which is unsafe if there are external DDLs finish between we reload the HMS schema and get the latest event id here. Those DDLs will be incorrectly skipped. To be safe, I think we should get the latest event id before L1522.
BTW, do we need this extra RPC call for non-hdfs tables, e.g. kudu table?
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1537
PS3, Line 1537: throw new CatalogException("Unable to load table metadata.", ex);
It seems too strict to fail the DDL just for failures in getting the latest event id. It'd be better to just log a warning about older events might not be skipped due to failures in fetching the latest event id.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1615
PS3, Line 1615: eventId = getCurrentEventId(msClient);
The same as above, I think this should run before the load() and exceptions in it should not fail the DDL.
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/3/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3219
PS3, Line 3219: setEnableSyncToLatestEventOnDdls(true)
Is this required to turn on this optimization?
http://gerrit.cloudera.org:8080/#/c/20022/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/3/tests/custom_cluster/test_events_custom_configs.py@316
PS3, Line 316: --enable_reload_events=true
Do we need this flag?
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Jul 2023 02:40:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#3).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
6 files changed, 140 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/3
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG@14
PS1, Line 14: eventTime
> Do we really compare the eventTime?
No, we actually compare with eventId. But for simplicity in talking terms I mentioned it as event time. Implementation wise we actually compare eventId.
http://gerrit.cloudera.org:8080/#/c/20022/1//COMMIT_MSG@19
PS1, Line 19: By comparing the lastRefreshEventId to current event id in the event
> How do we deal with batch events? For partition level INSERT/ALTER events,
Good point!! Addressed in the current patch set.
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1915
PS1, Line 1915: msPartition.getSd(), msPartition, permCache, partBuilder);
> please log this so we can reason about why an older event is skipped.
Done
http://gerrit.cloudera.org:8080/#/c/20022/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2883
PS1, Line 2883: * @param fileMetadataLoadOpts describes how to load file metadata
> Please log this so we can reason about why an older event is skipped.
Done
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Jul 2023 23:37:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 12:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2939
PS11, Line 2939: {} partiti
> nit: let's also log the number of reloaded partitions, e.g. partBuilderToPa
Agreed!!
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3224
PS11, Line 3224: testTblName);
: hiveExecutor.execute();
: alterTableAddParameter(testTblNam
> These are done in testInsertEvents(). I think we don't need them.
Ack
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS11, Line 3233: String testUnpartTb
> typo: "Even" -> "Event"
Done
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS11, Line 3233: fina
> nit: long
Ack
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3235
PS11, Line 3235: testInsert
> nit: use assertEquals() to have better error message
Ack
http://gerrit.cloudera.org:8080/#/c/20022/11/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3246
PS11, Line 3246: lastSyncEventIdBefore = testTbl.getLastRefreshEventId();
> Can we test that REFRESH will also update LastRefreshEventId? E.g. do some
The Java tests are not capable of making the event processor lagged. I'm using the Java tests only to check if lastRefreshEventId is updated or not. So, this tests are not really capable of checking skipped events. I have added end-to-end to simulate the same.
P.S: I added a sanity test for this ask.
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@324
PS11, Line 324: "create table {}.{} (i int) partitioned by (year int) "
> Can we also test transactional tables?
Ack
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@345
PS11, Line 345: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.run_stmt_in_hive(
: "insert into {db}.{tbl} partition (year=2024) values (10),(20),(30)"
: .format(db=unique_database, tbl=test_old_table_1))
: self.client.execute(
: "refresh {}.{}".format(unique_database, test_old_table_1))
> Could you double check these can finish in 5s? If not, some events might be
The test is running fine. I ran the test 10 times consecutively and it passed all the times.
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@363
PS11, Line 363:
> flake8: E122 continuation line missing indentation or outdented
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Aug 2023 04:55:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/11/tests/custom_cluster/test_events_custom_configs.py@363
PS11, Line 363: .
flake8: E122 continuation line missing indentation or outdented
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sun, 20 Aug 2023 19:25:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 16: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 16
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Aug 2023 04:56:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1527
PS4, Line 1527: reloadFileMetadata, reloadTableSchema, false, partitionsToUpdate,
: debugAction, partitionToEventId, reason);
> I am still trying to understand what will happen here. For example when cal
Fair point!! Let's update the lastRefreshEventId only when reloadFileMetadata and reloadTableSchema both are set to true.
I don't think having two different flags for tableSchema and fileMetadata at this point is useless. Because after IMPALA-10976, we are gonna combine creatEventId, lastRefreshEventId, LastSyncEventId e.t.c all these flags into one single flag.
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/4/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3233
PS4, Line 3233: "test", null);
: Long lastSyncEvenIdBefore = testTbl.getLastRefreshEventId();
: eventsProcessor_.processEvents();
> Wouldn't it be better to save the value of testTbl.getLastRefreshEventId()
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Aug 2023 17:04:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#10).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
set enable_skipping_older_events to true to enable this optimization
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
11 files changed, 201 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/10
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 9:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13757/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Aug 2023 20:11:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13760/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Aug 2023 23:01:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 14:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13837/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 14
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Aug 2023 23:21:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 13:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3229
PS12, Line 3229: alte
> nit: long
For Java tests, we can't delay the event processor, so at the event processor will process the events once they are generated.
I have add a condition just to verify that lastSyncEventId changes before and after.
http://gerrit.cloudera.org:8080/#/c/20022/12/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@3239
PS12, Line 3239: eventsProc
> nit: use assertEquals()
Ack. Sorry for the repetitions. I add it in the previous patch but some how it was lost.
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@374
PS12, Line 374: if is_partitoned:
> Can we also test for non-partitioned tables? We can extract a common method
Ack. After adding 4 set of sub tests, the events being skipped is becoming flaking, I tried varying polling interval but didn't help much. So I'll just be comparing if events_skipped_after is greater than events_skipped_before instead of comparing the exact number of skipped events.
http://gerrit.cloudera.org:8080/#/c/20022/12/tests/custom_cluster/test_events_custom_configs.py@384
PS12, Line 384:
> Let's verify no reloads happen by comparing the old and new metrics of "tab
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Aug 2023 20:14:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 13:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@323
PS13, Line 323: d
flake8: E306 expected 1 blank line before a nested definition, found 0
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@348
PS13, Line 348: e
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@353
PS13, Line 353: d
flake8: E501 line too long (93 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@360
PS13, Line 360: e
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/20022/13/tests/custom_cluster/test_events_custom_configs.py@372
PS13, Line 372: d
flake8: E501 line too long (93 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Aug 2023 20:16:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/14/tests/custom_cluster/test_events_custom_configs.py@337
PS14, Line 337: partitions_refreshed_before = \
> Can we also verify no table/partitions are refreshed here? We can extract t
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Aug 2023 17:04:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 15
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Aug 2023 00:40:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Jun 2023 00:22:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13719/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 03:32:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13755/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Aug 2023 19:42:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1491
PS6, Line 1491: if (isOlderEvent(null)) {
: infoLog("Not processing the alter table event {} as it is an older event",
: getEventId());
: return;
: }
:
: if (isRename_) {
: processRename();
: return;
: }
I am trying to understand the RENAME case. I think that it would be clearer to skip isOlderEvent() logic for RENAME, as renameTableFromEvent() has its own handling of edge cases like renaming a table and then creating a table with the same name.
http://gerrit.cloudera.org:8080/#/c/20022/6/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2341
PS6, Line 2341: isOlderEvent
I am trying to understand what will happen in the batched case.
Wouldn't it make sense to call isOlderEvent for batchedEvents_ individually? If I understand correctly, currently we consider the eventId of the last event in batchedEvents_, so if the refresh happened somewhere in the middle, then we won't skip the early events. For example:
from Hive: insert to partition of tbl
from Impala: refresh tbl
from Hive: insert to partition of tbl
The events from the two events could be batched, right?
Also, this path looks untested as we always do only a single insert in the tests. This could be covered by an insert with dynamic partitioning in Hive.
http://gerrit.cloudera.org:8080/#/c/20022/6/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/6/tests/custom_cluster/test_events_custom_configs.py@315
PS6, Line 315: catalogd_args
Is there a reason why enable_sync_to_latest_event_on_ddls is true in the Java test but the default false here?
Is the long term goal to enable that flag by default?
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Aug 2023 16:01:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20022
to look at the new patch set (#2).
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
IMPALA-11535: Skip older events in the event processor based on the
latestRefreshEventID
Summary: If the table has been manually refreshed, all its events
happen before the manual REFRESH can be skipped. This happens when
catalogd is lagging behind in processing events. When processing an
event, we can check whether there are manual REFRESH executed after
its eventTime. In such case, we don't need to process the event to
refresh anything. This helps catalogd to catch up HMS events quickly.
Implementation details: Updated the lastRefreshEventId on the table or
partition whenever there is table or partition level refresh/load.
By comparing the lastRefreshEventId to current event id in the event
processor the older events can be skipped.
Testing:
- Unit end-to-end test and unit test to test the functionality.
Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
6 files changed, 140 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/20022/2
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20022 )
Change subject: IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID
......................................................................
Patch Set 10:
(2 comments)
With the flag default as false the change seems safe to me. Thanks for the work!
Few more comments about the tests.
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py@348
PS7, Line 348: insert into
Can you also add a test with dynamic partitioning?
http://gerrit.cloudera.org:8080/#/c/20022/7/tests/custom_cluster/test_events_custom_configs.py@357
PS7, Line 357: tbl_events_skipped_before
Can't we use fixed numbers in the test, e.g. assert tbl_events_skipped_after == tbl_events_skipped_before + 3? I don't know how many events we will skip exactly, but it seems possible that other skippable events are also triggered besides the inserts.
--
To view, visit http://gerrit.cloudera.org:8080/20022
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1
Gerrit-Change-Number: 20022
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Aug 2023 10:21:56 +0000
Gerrit-HasComments: Yes