You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2025/08/26 14:13:23 UTC
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23349
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
IMPALA-14358: Event processing can invalidate Iceberg tables
CatalogServiceCatalog.reloadTableIfExists() can throw a
ClassCastException when it reloads an Iceberg table. When it happens
during event processing the event processor invalidates the table.
This happens usually happens when another engine updates an Iceberg
table. It then causes slow table loading times as the tables need to
be fully reloaded instead of just doing an incremental table loading.
This patches fixes the ClassCastException by moving the cast into an
if statement.
Testing
* e2e tests added
Change-Id: I892cf326a72024674facad6750893352b982c658
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/metadata/test_event_processing.py
2 files changed, 42 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/23349/1
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG@9
PS1, Line 9: CatalogServiceCatalog.reloadTableIfExists() can throw a
: ClassCastException when it reloads an Iceberg table
> This looks like a fresh change - can you mention the commit that broke it a
Good to know it's a fresh change. Mentioned it in the comment and linked the issues together.
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:26:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG@9
PS1, Line 9: IMPALA-12829 added extra code to CatalogServiceCatalog's
: reloadTableIfExists() that can throw a ClassCastExc
> Good to know it's a fresh change. Mentioned it in the comment and linked th
oops, I was in that review
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:35:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG@9
PS1, Line 9: CatalogServiceCatalog.reloadTableIfExists() can throw a
: ClassCastException when it reloads an Iceberg table
This looks like a fresh change - can you mention the commit that broke it and also connect the Jiras? This would help in avoiding creating releases with the problematic commit but without the fix.
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:20:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20273/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:38:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 19:41:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12379/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:41:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
IMPALA-14358: Event processing can invalidate Iceberg tables
IMPALA-12829 added extra code to CatalogServiceCatalog's
reloadTableIfExists() that can throw a ClassCastException when it
reloads an Iceberg table. When it happens during event processing
the event processor invalidates the table.
This usually happens when another engine updates an Iceberg table. It
then causes slow table loading times as the tables need to be fully
reloaded instead of just doing an incremental table loading.
This patches fixes the ClassCastException by moving the cast into an
if statement.
Testing
* e2e tests added
Change-Id: I892cf326a72024674facad6750893352b982c658
Reviewed-on: http://gerrit.cloudera.org:8080/23349
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/metadata/test_event_processing.py
2 files changed, 42 insertions(+), 25 deletions(-)
Approvals:
Csaba Ringhofer: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23349/1//COMMIT_MSG@9
PS1, Line 9: IMPALA-12829 added extra code to CatalogServiceCatalog's
: reloadTableIfExists() that can throw a ClassCastExc
> oops, I was in that review
Even the best ones make mistakes sometimes :)
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:40:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23349
to look at the new patch set (#2).
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
IMPALA-14358: Event processing can invalidate Iceberg tables
IMPALA-12829 added extra code to CatalogServiceCatalog's
reloadTableIfExists() that can throw a ClassCastException when it
reloads an Iceberg table. When it happens during event processing
the event processor invalidates the table.
This usually happens when another engine updates an Iceberg table. It
then causes slow table loading times as the tables need to be fully
reloaded instead of just doing an incremental table loading.
This patches fixes the ClassCastException by moving the cast into an
if statement.
Testing
* e2e tests added
Change-Id: I892cf326a72024674facad6750893352b982c658
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/metadata/test_event_processing.py
2 files changed, 42 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/23349/2
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14358: Event processing can invalidate Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23349 )
Change subject: IMPALA-14358: Event processing can invalidate Iceberg tables
......................................................................
Patch Set 1:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23349/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/23349/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3422
PS1, Line 3422: // Transactional table events using this method should reload the table only if the
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/23349
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I892cf326a72024674facad6750893352b982c658
Gerrit-Change-Number: 23349
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 14:15:43 +0000
Gerrit-HasComments: Yes