You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2024/01/11 22:25:37 UTC
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20887
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated one impala cluster is consumed bu other impala clusters.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 23 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/1
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:34:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10176/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jan 2024 01:49:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5: Code-Review+1
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1094
PS3, Line 1094: self.execute_query("create database if not exists {0}".format(unique_database))
> Yeah agreed. Test will take care of dropping the DB after the test is done.
I still don't get it - as the unique_database fixture always creates the database with the same name this line should be always NOOP
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1099
PS3, Line 1099: " ".join
> L#335 -- verify_skipping_older_events() also uses similar syntax and fewer
ack
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1101
PS3, Line 1101: self.__get_transactional_tblproperties(is_transactional)])
> L#1100, L#1101 are values separated by ',' in the same array. So I don't th
Most of this file seems to use the new block: 2 space, line break: 4 space scheme, but I don't want to be pushy about it :) Unlike c++, the style guide is not explicit about Python code https://cwiki.apache.org/confluence/display/IMPALA/Impala+Style+Guide
http://gerrit.cloudera.org:8080/#/c/20887/5/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/5/tests/custom_cluster/test_events_custom_configs.py@1111
PS5, Line 1111: is_batched
The goal of is_batched is not clear to me - is it enough if truncate and the next insert are processed in the same batch, or the previous inserts should be also processed in the same batch? In the latter case line 1105/1106 should be skipped in the batched case.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 17:15:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14996/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:56:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#2).
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 72 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/2
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#6).
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 102 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/6
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20887/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-12636
> IMPALA-12636 is a follow-up task of IMPALA-10976. Can we file a new JIRA fo
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1563
PS3, Line 1563: Stats may not be computed on a table when a
: // truncate operation is executed, so return only when it is not a truncate op.
> nit: This is not that clear for me. Not all parameters we ignore in canBeSk
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2218
PS3, Line 2218: Preconditions.checkNotNull
> nit: Don't need checkNotNull() since getIsTruncateOp() returns the primitiv
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2294
PS3, Line 2294: !isTruncateOp_
> nit: same as L1565, moving this into canBeSkipped() seems cleaner.
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2481
PS3, Line 2481: // force reload truncated partition events
> nit: catalogd reloads the partitions, not the partition events
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1094
PS3, Line 1094: self.execute_query("create database if not exists {0}".format(unique_database))
> This shouldn't be necessary with unique_database
Yeah agreed. Test will take care of dropping the DB after the test is done. If we were to create a custom database in the test, then we need to handle the drop logic. So it's just for ease of use that I'm using this way.
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1099
PS3, Line 1099: " ".join
> Using join here looks strange to me - shouldn't we simply use the format st
L#335 -- verify_skipping_older_events() also uses similar syntax and fewer lines of code/variables than using format string. Let me know if is still difficult for readability purposes, I can change and implement your suggestion.
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1101
PS3, Line 1101: self.__get_transactional_tblproperties(is_transactional)])
> nit: +2 indentation on line breaks
L#1100, L#1101 are values separated by ',' in the same array. So I don't think we need indentation here.
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1105
PS3, Line 1105: self.run_stmt_in_hive(insert_query.format(unique_database, tbl_name))
> as the test's goal is to check handling of truncate event, shouldn't we wai
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1111
PS3, Line 1111: (1),(2),(3)
> There are also 3 values before truncate. Can we use 2 or 4 values here? So
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1123
PS3, Line 1123: verify_truncate_op_in_hive(hive_tbl, True, True, True)
> Can we add one more test case that truncate more than one partitions? e.g.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Jan 2024 20:42:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 23:53:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1094
PS3, Line 1094: self.execute_query("create database if not exists {0}".format(unique_database))
> I still don't get it - as the unique_database fixture always creates the da
Ack
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1101
PS3, Line 1101: self.__get_transactional_tblproperties(is_transactional)])
> Most of this file seems to use the new block: 2 space, line break: 4 space
Ack
http://gerrit.cloudera.org:8080/#/c/20887/5/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/5/tests/custom_cluster/test_events_custom_configs.py@1111
PS5, Line 1111: is_batched
> The goal of is_batched is not clear to me - is it enough if truncate and th
The goal is to verify that I wanted to verify a test scenario where truncate and insert partitions are batched together.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:21:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14948/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 20:08:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20887/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/2/tests/custom_cluster/test_events_custom_configs.py@1096
PS2, Line 1096: d
flake8: E306 expected 1 blank line before a nested definition, found 0
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 19:38:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20887/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-12636
IMPALA-12636 is a follow-up task of IMPALA-10976. Can we file a new JIRA for this bug?
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1472
PS3, Line 1472: Preconditions.checkNotNull
nit: Don't need checkNotNull() since getIsTruncateOp() returns the primitive boolean type.
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1563
PS3, Line 1563: Stats may not be computed on a table when a
: // truncate operation is executed, so return only when it is not a truncate op.
nit: This is not that clear for me. Not all parameters we ignore in canBeSkipped() are stats: transient_lastDdlTime, totalSize, numFilesErasureCoded, numFiles.
What about moving the check on 'isTruncateOp_' into canBeSkipped() and update the comment there, saying events from truncate ops always can't be skipped?
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2218
PS3, Line 2218: Preconditions.checkNotNull
nit: Don't need checkNotNull() since getIsTruncateOp() returns the primitive boolean type.
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2294
PS3, Line 2294: !isTruncateOp_
nit: same as L1565, moving this into canBeSkipped() seems cleaner.
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1111
PS3, Line 1111: (1),(2),(3)
There are also 3 values before truncate. Can we use 2 or 4 values here? So we can be sure at L1114 that we are reading the new values.
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1123
PS3, Line 1123: verify_truncate_op_in_hive(hive_tbl, True, True, True)
Can we add one more test case that truncate more than one partitions? e.g.
impala> create table truncate_part(i int) partitioned by (p int, q int);
impala> insert into truncate_part partition(p, q) values (0,0,0), (0,0,1), (0,0,2); // (p=0,q=0,1,2)
impala> insert into truncate_part partition(p, q) values (0,1,0), (0,1,1); // (p=1, q=0,1)
hive> truncate table truncate_part partition(p=0); // then verify there are only 2 rows in Impala
hive> truncate table truncate_part; // then verify no rows left in Impala
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Jan 2024 13:47:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#4).
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 96 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/4
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG@15
PS1, Line 15: Note: Alter table event for an external table generated by the truncate
: operation from Impala cannot be identified if it's a truncate op or not
Is it only true on tables that are not being replicated? For tables that are being replicated, Impala does invoke the HMS truncateTable API so events will be generated:
https://github.com/apache/impala/blob/a2b8aed2c2d4c1bb25ed9626a2b014b79ec741ad/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L3314-L3318
Or are those events don't set the isTruncateOp field?
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG@18
PS1, Line 18: generated one impala cluster is consumed bu other impala clusters.
nit: "generated" -> "generated from"
"bu" -> "by"
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1565
PS1, Line 1565: if (canBeSkipped() && !isTruncateOp_) {
nit: check 'isTruncateOp_' first since it's more light-weight.
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1570
PS1, Line 1570: skipFileMetadataReload_ = canSkipFileMetadataReload(tableBefore_, tableAfter_)
: && !isTruncateOp_;
nit: check 'isTruncateOp_' first since it's more light-weight.
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2207
PS1, Line 2207: AlterPartitionMessage alterPartitionMessage =
What about partition level truncate operations? AlterPartitionMessage also has the isTruncateOp field.
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py@1096
PS1, Line 1096: create table {0}.{1} (i int)""".format(unique_database, tbl_name))
Can we also test on partitioned tables and transactional tables?
Also test on partition-level truncate: TRUNCATE [TABLE] table_name [PARTITION partition_spec];
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 23:55:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14947/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 20:05:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14965/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Jan 2024 21:09:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/3/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2481
PS3, Line 2481: // force reload truncated partition events
nit: catalogd reloads the partitions, not the partition events
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1094
PS3, Line 1094: self.execute_query("create database if not exists {0}".format(unique_database))
This shouldn't be necessary with unique_database
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1099
PS3, Line 1099: " ".join
Using join here looks strange to me - shouldn't we simply use the format string? e.g.
partitioned_str = " partitioned by (year int) " if is_partitioned else ''
create_query = "create table `{}`.`{}` (i int) {} {}".format(unique_database, tbl_name, partitioned_str , values)
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1101
PS3, Line 1101: self.__get_transactional_tblproperties(is_transactional)])
nit: +2 indentation on line breaks
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1105
PS3, Line 1105: self.run_stmt_in_hive(insert_query.format(unique_database, tbl_name))
as the test's goal is to check handling of truncate event, shouldn't we wait here for the event processor to process the insert event?
there are 2 cases we should avoid to ensure that the test provides proper coverage:
1. the table is reloaded, but not because of the truncate event, but the insert event - as the reload happens after the truncate already happened, catalogd would see an empty table during reload
2. both events are ignored because the table is an IncompleteTable in the catalogd, and the table load is initiated because of the select query, not the event processor
The problem with 1 and 2 is that the test would pass even without the truncate event.
So actually I think that the best way is to call a REFRESH in Impala before the truncate to ensure that the state with 3 rows is cached and the truncate event handling is really needed to trigger the reload
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Jan 2024 12:31:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jan 2024 06:22:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 8:
> Patch Set 8: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10173/
The failure is unrelated and is tracked in IMPALA-11166
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jan 2024 01:49:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1724
PS4, Line 1724: if (isTruncateOp_) {
> nit: The work of deepCopy() can be saved if isTrucateOp_ == true. Let's che
Ack
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2343
PS4, Line 2343: uncateOp_) {
> nit: let's check this at the begining of the method
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 03:24:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14975/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 03:52:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10165/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 19:21:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#7).
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 101 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/7
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10173/
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jan 2024 01:08:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py@1102
PS1, Line 1102: u
flake8: E122 continuation line missing indentation or outdented
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 22:26:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jan 2024 01:49:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Reviewed-on: http://gerrit.cloudera.org:8080/20887
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 101 insertions(+), 10 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14995/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:46:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#3).
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 73 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/3
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 4: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1724
PS4, Line 1724: org.apache.hadoop.hive.metastore.api.Table tblAfter = tableAfter_.deepCopy();
nit: The work of deepCopy() can be saved if isTrucateOp_ == true. Let's check it at the beginning of this method.
if (isTruncateOp_) return false;
http://gerrit.cloudera.org:8080/#/c/20887/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2343
PS4, Line 2343: isTruncateOp_
nit: let's check this at the begining of the method
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 02:41:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jan 2024 04:15:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Jan 2024 22:52:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG@15
PS1, Line 15: Note: Alter table event for an external table generated by the truncate
: operation from Impala cannot be identified if it's a truncate op or not
> Is it only true on tables that are not being replicated? For tables that ar
I forgot to mention about replicated tables. Truncate on replicated tables have isTruncateOp field being set since the event is generated by HMS.
http://gerrit.cloudera.org:8080/#/c/20887/1//COMMIT_MSG@18
PS1, Line 18: generated from one impala cluster is consumed by other impala clusters.
> nit: "generated" -> "generated from"
Ack
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1565
PS1, Line 1565: if (!isTruncateOp_ && canBeSkipped()) {
> nit: check 'isTruncateOp_' first since it's more light-weight.
Ack
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1570
PS1, Line 1570: skipFileMetadataReload_ = !isTruncateOp_ && canSkipFileMetadataReload(tableBefore_,
: tableAfter_);
> nit: check 'isTruncateOp_' first since it's more light-weight.
Ack
http://gerrit.cloudera.org:8080/#/c/20887/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@2207
PS1, Line 2207: Preconditions.checkState(getEventType().equals(MetastoreEventType.ALTER_PARTITION));
> What about partition level truncate operations? AlterPartitionMessage also
Ack
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/1/tests/custom_cluster/test_events_custom_configs.py@1096
PS1, Line 1096:
> Can we also test on partitioned tables and transactional tables?
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 19:40:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12636: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1096
PS3, Line 1096:
flake8: W293 blank line contains whitespace
http://gerrit.cloudera.org:8080/#/c/20887/3/tests/custom_cluster/test_events_custom_configs.py@1096
PS3, Line 1096:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 12 Jan 2024 19:41:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20887
to look at the new patch set (#5).
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
When the event processor receives an alter table event and the event
type is of truncate operation currently file metadata is not reloaded.
This patch addresses this issue, where the alter table event type is
verified if it's a truncate operation and then reload the file metadata
accordingly.
Note: Alter table event for an external table generated by the truncate
operation from Impala cannot be identified if it's a truncate op or not
This becomes an issue in multi cluster Impala environments where events
generated from one impala cluster is consumed by other impala clusters.
Truncate operations in Impala on replicated tables will generated alter
event with 'isTruncateOp' field set to true.
Testing:
Added an end-to-end test to verify whether file metadata is reloaded
for the above scenario.
Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 102 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/20887/5
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:35:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20887 )
Change subject: IMPALA-12719: Reload filemetadata for AlterTable event of type truncate
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10173/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20887
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I53bb80c294623eec7c79d9f30f410771386c6b75
Gerrit-Change-Number: 20887
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jan 2024 20:35:28 +0000
Gerrit-HasComments: No