You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2024/08/12 19:39:46 UTC
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21662
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
IMPALA-12758: Event processor is ignoring the prev_id while reloading
existing partitions
IMPALA-10976 depends on event processor for cache consistency instead
of directly modifying the cache inplace. so when
enable_sync_to_latest_event_on_ddls is enabled, cache modification via
partitioned table events should retain the prev_id of the partitions
while reloading the partitions. Currently, prev_id is set to -1, so
when the catalogDelta is sent from the state store to impala demons,
because prev_id is not valid, impalads will not know whether to
invalidate the current partition and then request the new partition
information.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an ened-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 21 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/1
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 2:
(5 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
flake8: W605 invalid escape sequence '\['
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
flake8: W605 invalid escape sequence '\d'
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
flake8: W605 invalid escape sequence '\('
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
flake8: W605 invalid escape sequence '\)'
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
flake8: W605 invalid escape sequence '\]'
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Aug 2024 19:34:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11073/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Oct 2024 02:28:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 6: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG@12
PS6, Line 12: impalads
nit: To be clear, I think "LocalCatalog mode coordinators" or "coordinators in LocalCatalog mode" is better.
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG@18
PS6, Line 18: This is only an issue in local catalog mode
nit: Please add "for REFRESH <partition> or updating partitions based on HMS events"
http://gerrit.cloudera.org:8080/#/c/21662/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3094
PS6, Line 3094: if (entry.getValue() != null) {
: dropPartition(entry.getValue(), false);
: }
: addPartition(entry.getKey().build());
We always drop and add the partition instance no matter whether the partition metadata is actually updated. So REFRESH <partition> will always update the partition. That's different to REFRESH <table> where we skip unchanged partitions here:
https://github.com/apache/impala/blob/42fda24364786cc1a457890bd212bb3922479e95/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1098-L1101
Filed IMPALA-13453 to track this issue.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 02:18:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#4).
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
IMPALA-12758: Event processor is ignoring the prev_id while reloading
existing partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the state
store to impala demons, because prev_id is not valid, impalads will
not know whether to invalidate the current partition and then request
the new partition information.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode or when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an end-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 27 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/4
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@876
PS4, Line 876: /**
: * Returns a string of the form part_key1=value1/part_key2=value2...
:
> Do we use this somewhere?
No, it is not being used after the latest patchset. I'll remove it.
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3036
PS4, Line 3036: public void reloadPartitions(IMetaStoreClient client,
> If you check the call graphs of this method, you'll see it's not just used
Ack, Added a test for the same.
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1310
PS1, Line 1310: @pytest.mark.execute_serially
> Should we test on local-catalog mode? 'prev_id' of HdfsPartition is not use
Ack
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: self.client.execute("select * from {}".format(test_tbl))
> Post an idea of how to add a test in another comment
In the local catalog mode, I have created a partitioned table and loaded the table metadata in the catalog and coordinator. Then I inserted some partition values from Hive and when the event processor processed these partitions, I verified that stale partition metadata is invalidated in the coordinator by looking at the coordinator logs.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 18:35:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#5).
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
IMPALA-12758: Event processor is ignoring the prev_id while reloading
existing partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the state
store to impala demons, because prev_id is not valid, impalads will
not know whether to invalidate the current partition and then request
the new partition information.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode or when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an end-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_local_catalog.py
3 files changed, 38 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/5
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16734/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Aug 2024 20:05:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Aug 2024 21:11:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#3).
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
IMPALA-12758: Event processor is ignoring the prev_id while reloading
existing partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the state
store to impala demons, because prev_id is not valid, impalads will
not know whether to invalidate the current partition and then request
the new partition information.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode or when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an ened-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 27 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/3
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#2).
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
IMPALA-12758: Event processor is ignoring the prev_id while reloading
existing partitions
IMPALA-10976 depends on event processor for cache consistency instead
of directly modifying the cache inplace. so when
enable_sync_to_latest_event_on_ddls is enabled, cache modification via
partitioned table events should retain the prev_id of the partitions
while reloading the partitions. Currently, prev_id is set to -1, so
when the catalogDelta is sent from the state store to impala demons,
because prev_id is not valid, impalads will not know whether to
invalidate the current partition and then request the new partition
information.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an ened-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 27 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/2
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16909/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Sep 2024 23:43:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the
statestore to impala demons, because prev_id is not valid, coordinators
in local catalog mode will not know whether to invalidate the current
partition.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode for
REFRESH <partition> or updating partitions based on HMS events.
Testing:
- Added an end-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Reviewed-on: http://gerrit.cloudera.org:8080/21662
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_local_catalog.py
3 files changed, 40 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Oct 2024 02:28:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16663/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 20:11:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG@12
PS6, Line 12: coordina
> nit: To be clear, I think "LocalCatalog mode coordinators" or "coordinators
Ack
http://gerrit.cloudera.org:8080/#/c/21662/6//COMMIT_MSG@18
PS6, Line 18:
> nit: Please add "for REFRESH <partition> or updating partitions based on HM
Ack
http://gerrit.cloudera.org:8080/#/c/21662/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/6/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3094
PS6, Line 3094: if (entry.getValue() != null) {
: dropPartition(entry.getValue(), false);
: }
: addPartition(entry.getKey().build());
> We always drop and add the partition instance no matter whether the partiti
Ack. We update the file metadata before this logic. So we can really skip drop/create if the partitions are same.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Oct 2024 17:52:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17186/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Oct 2024 20:38:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: self.client.execute("select * from {}".format(test_tbl))
> The bug is there without the patch of IMPALA-10976. I think we should be ab
Ack
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326:
> use the raw string format to get rid of these warnings, i.e.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Aug 2024 20:41:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Oct 2024 07:48:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 28 Oct 2024 02:27:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 5: Code-Review+1
(7 comments)
The new test looks good to me. Just have some minor comments.
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@7
PS5, Line 7: IMPALA-12758: Event processor is ignoring the prev_id while reloading
Please update the commit message. This is not only used by the event-processor. Maybe "Fix catalogd not setting prev_id for reloaded partitions" is better.
The commit message also needs to be updated.
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@12
PS5, Line 12: state
: store
nit: "statestore"
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@13
PS5, Line 13: impalads will
: not know whether to invalidate the current partition and then request
: the new partition information.
It's unclear to me what do we mean by "request the new partition information". Coordinator won't send requests when processing statestore updates.
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: self.client.execute("select * from {}".format(test_tbl))
> In the local catalog mode, I have created a partitioned table and loaded th
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@580
PS5, Line 580: hms_event_polling_interval_s=5
If the default interval doesn't work, let's set this to 0 to avoid flakiness.
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@587
PS5, Line 587: self.client.execute("show partitions {}".format(test_tbl))
Please add a comment to explain the purpose of this, e.g. "Make the partition loaded in the coordinator, so the one instance needs to be invalidated when the partition is reloaded."
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@589
PS5, Line 589: log_regex = r"Invalidated objects in cache: \[partition %s:p=\d \(id=%%d\)\]" \
nit: change "p=\d" to "p=0" and change "%%d" to "0"
Then we can change "log_regex % 0" to "log_regex"
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Oct 2024 00:14:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#7).
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the
statestore to impala demons, because prev_id is not valid, coordinators
in local catalog mode will not know whether to invalidate the current
partition.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode for
REFRESH <partition> or updating partitions based on HMS events.
Testing:
- Added an end-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_local_catalog.py
3 files changed, 40 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/7
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java:
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@876
PS4, Line 876: public long getPrevId() {
: return prevId_;
: }
Do we use this somewhere?
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3036
PS4, Line 3036: public void reloadPartitions(IMetaStoreClient client,
If you check the call graphs of this method, you'll see it's not just used by the event-processor, but also used by REFRESH with partition spec.
HdfsTable.reloadPartitions(IMetaStoreClient, Map<Partition, HdfsPartition>, boolean, EventSequence)(2 usages) (org.apache.impala.catalog)
CatalogServiceCatalog.reloadHdfsPartition(HdfsTable, String, Reference<Boolean>, ThriftObjectType, String, long, HdfsPartition, ...) (org.apache.impala.catalog)
CatalogServiceCatalog.reloadPartition(Table, List<TPartitionKeyValue>, Reference<Boolean>, ThriftObjectType, String, EventSequence) (org.apache.impala.catalog)
CatalogOpExecutor.execResetMetadataImpl(TResetMetadataRequest) (org.apache.impala.service)
CatalogOpExecutor.execResetMetadata(TResetMetadataRequest) (org.apache.impala.service)
So we can add a test on REFRESH with partition spec, e.g. REFRESH functional.alltypes PARTITION (year=2009, month=1).
The test should run in local catalog mode:
* SHOW PARTITIONS functional.alltypes - make sure all partitions are loaded in both catalogd and coordinator side
* REFRESH functional.alltypes PARTITION (year=2009, month=1)
* Wait 2s and see if there are logs like
CatalogdMetaProvider.java:1680] Invalidated objects in cache: [partition functional.alltypes:year=2009/month=1 (id=xxx)]
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: self.client.execute("select * from {}".format(test_tbl))
> Ack
Post an idea of how to add a test in another comment
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 Sep 2024 08:06:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21662/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/3//COMMIT_MSG@24
PS3, Line 24: ened
typo
http://gerrit.cloudera.org:8080/#/c/21662/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/3/tests/custom_cluster/test_events_custom_configs.py@1324
PS3, Line 1324: self.client.execute("select * from {}".format(test_tbl))
What is the goal of this select *? To load all partitions? Does it have to happen before processing the events from the insert (if yes, then this looks a bit fragile with hms_event_polling_interval_s=1)?
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Sep 2024 15:12:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21662/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/3//COMMIT_MSG@24
PS3, Line 24: end-
> typo
Ack
http://gerrit.cloudera.org:8080/#/c/21662/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/3/tests/custom_cluster/test_events_custom_configs.py@1324
PS3, Line 1324: self.client.execute("select * from {}".format(test_tbl))
> What is the goal of this select *? To load all partitions? Does it have to
No, I just want to load the table in cache, so that events will not be ignored at L#1325. I'll bump up the event_polling interval just to be on the safe side.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Sep 2024 23:12:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@7
PS5, Line 7: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
> Please update the commit message. This is not only used by the event-proces
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@12
PS5, Line 12: ads
: will
> nit: "statestore"
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5//COMMIT_MSG@13
PS5, Line 13: ion.
:
: This patch addresses this issu
> It's unclear to me what do we mean by "request the new partition informatio
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py
File tests/custom_cluster/test_local_catalog.py:
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@580
PS5, Line 580: hms_event_polling_interval_s=0
> If the default interval doesn't work, let's set this to 0 to avoid flakines
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@587
PS5, Line 587: # Make the partition loaded in the coordinator, so that one instance needs to be
> Please add a comment to explain the purpose of this, e.g. "Make the partiti
Ack
http://gerrit.cloudera.org:8080/#/c/21662/5/tests/custom_cluster/test_local_catalog.py@589
PS5, Line 589: self.client.execute("show partitions {}".format(test_tbl))
> nit: change "p=\d" to "p=0" and change "%%d" to "0"
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Oct 2024 20:13:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17083/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 19:01:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: EventProcessorUtils.wait_for_event_processing(self)
> This patch is valid when enable_sync_to_latest_event_on_ddls=true, Since IM
The bug is there without the patch of IMPALA-10976. I think we should be able to add a test. Or could you explain why the bug can't be reproduced by a test?
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/2/tests/custom_cluster/test_events_custom_configs.py@1326
PS2, Line 1326: \
> flake8: W605 invalid escape sequence '\('
use the raw string format to get rid of these warnings, i.e.
r"Invalidated objects in cache: \[partition %s:p=\d \(id=%%d\)\]"
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Aug 2024 07:21:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21662/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3019
PS1, Line 3019: ument(oldPar
> Based on L3020 and L3023, 'oldPartition' could be null.
Ack
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: EventProcessorUtils.wait_for_event_processing(self)
> This test passes without the fix. Can we add one that will fail without the
This patch is valid when enable_sync_to_latest_event_on_ddls=true, Since IMPALA-10976 is still in review, I cannot add a test case that works only with this patch. So, either we'll have wait until IMPALA-10976 gets merged or proceed forward without a test case since it is a one-liner change.
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Aug 2024 19:35:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21662/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/21662/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@3019
PS1, Line 3019: oldPartition
Based on L3020 and L3023, 'oldPartition' could be null.
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1310
PS1, Line 1310: @CustomClusterTestSuite.with_args(catalogd_args="--hms_event_polling_interval_s=5")
Should we test on local-catalog mode? 'prev_id' of HdfsPartition is not used in ImpaladCatalog (i.e. legacy catalog mode).
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1324
PS1, Line 1324: self.assert_impalad_log_contains('INFO', log_regex.format(3, 3, 3))
This test passes without the fix. Can we add one that will fail without the fix?
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Aug 2024 11:51:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Event processor is ignoring the prev id while reloading existing partitions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21662 )
Change subject: IMPALA-12758: Event processor is ignoring the prev_id while reloading existing partitions
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21662/1/tests/custom_cluster/test_events_custom_configs.py@1322
PS1, Line 1322: \
flake8: W605 invalid escape sequence '\,'
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Aug 2024 19:40:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12758: Fix catalogd not setting prev id for reloaded partitions
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21662
to look at the new patch set (#6).
Change subject: IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
......................................................................
IMPALA-12758: Fix catalogd not setting prev_id for reloaded partitions
Cache modification via partitioned table events should retain the
prev_id of the partitions while reloading the partitions. Currently,
prev_id is set to -1, so when the catalogDelta is sent from the
statestore to impala demons, because prev_id is not valid, impalads
will not know whether to invalidate the current partition.
This patch addresses this issue by setting the prev_id of reloaded
partition.
Note: This is only an issue in local catalog mode or when
enable_sync_to_latest_event_on_ddls is enabled
Testing:
- Added an end-to-end test to verify the same
Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_local_catalog.py
3 files changed, 40 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/21662/6
--
To view, visit http://gerrit.cloudera.org:8080/21662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia7d53f601b63d83e99cbd2d1f58076a0dc78c610
Gerrit-Change-Number: 21662
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>