You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org> on 2023/09/15 00:44:30 UTC
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Sai Hemanth Gantasala has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20485
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition
events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 46 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/1
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10031/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 04:54:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20485
to look at the new patch set (#2).
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition
events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 43 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1282
PS1, Line 1282: n
If we already check getLastSyncedEventId(), wouldn't it be simpler to drop the event before creating batches?
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1284
PS1, Line 1284: try {
: org.apache.impala.catalog.Table tbl = catalog_.getTable(dbName_, tblName_);
: if (tbl != null && tbl.getLastSyncedEventId() >= event.getEventId()) {
: return false;
: }
: } catch (CatalogException e) {
: debugLog("ignoring exception while fetching from cache on table {}.{}",
: dbName_, tblName_, e);
: }
This seems to be the same as for alter table events - a function like bool MetastoreEvent.newerThanLastSyncedEventId() could be created to reduce duplication.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Dec 2023 19:51:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Reviewed-on: http://gerrit.cloudera.org:8080/20485
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 58 insertions(+), 13 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 13
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Dec 2023 23:54:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10020/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 13:06:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10027/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 18:54:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14007/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 1
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Sep 2023 01:08:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 00:21:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@741
PS2, Line 741: Newer
the name doesn't match the semantics, the function returns true if the event's event id is NOT newer than lastSyncedEventId
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@747
PS2, Line 747: catch
Exception handling could be skipped by using getTableNoThrow instead of getTable. It returns null if the db or table is not found.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 09:06:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20485/4/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20485/4/tests/custom_cluster/test_events_custom_configs.py@409
PS4, Line 409: assert current_skipped_events > prev_skipped_events
This only verifies there are events being skipped but not all the events. There are 24 partitions in total. Can we verify the counter bump >= 24?
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 08:34:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 08:33:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20485
to look at the new patch set (#4).
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 41 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/4
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2023 00:19:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 19:49:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20485/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-10949: Improve batching logic of partition events
:
> nit: put the title in one line
Ack
http://gerrit.cloudera.org:8080/#/c/20485/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20485/3/tests/custom_cluster/test_events_custom_configs.py@403
PS3, Line 403: self.client.execute("refresh {0}.{1}".format(unique_database, test_batch_table))
> Can we also verify all events are skipped? The metric of "events-skipped" c
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Dec 2023 20:28:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20485
to look at the new patch set (#5).
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 41 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/5
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 5: Code-Review+2
Carrying Csaba's +1.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 23:36:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 23:36:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 6:
> Patch Set 6: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9999/
There is a FE test failure: https://jenkins.impala.io/job/ubuntu-20.04-from-scratch/1079
java.lang.AssertionError: expected:<1> but was:<10>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at org.apache.impala.catalog.events.MetastoreEventsProcessorTest.runEventBatchingTest(MetastoreEventsProcessorTest.java:2466)
at org.apache.impala.catalog.events.MetastoreEventsProcessorTest.testEventBatching(MetastoreEventsProcessorTest.java:2452)
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Dec 2023 08:33:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9:
> Patch Set 9: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10023/
This seems a flaky test tracked by IMPALA-12416:
custom_cluster/test_events_custom_configs.py:375: in test_skipping_older_events
verify_skipping_older_events(test_old_table, False, False)
custom_cluster/test_events_custom_configs.py:355: in verify_skipping_older_events
query.format(unique_database, table_name), table_name)
custom_cluster/test_events_custom_configs.py:342: in verify_skipping_hive_stmt_events
assert tbl_events_skipped_after > tbl_events_skipped_before
E assert 1 > 1
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 00:21:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@741
PS2, Line 741: Older
> the name doesn't match the semantics, the function returns true if the even
Ack
http://gerrit.cloudera.org:8080/#/c/20485/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@747
PS2, Line 747:
> Exception handling could be skipped by using getTableNoThrow instead of get
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 19:33:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14627/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 4
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Dec 2023 20:56:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 3: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20485/3//COMMIT_MSG@7
PS3, Line 7: IMPALA-10949: Improve batching logic of partition
: events
nit: put the title in one line
http://gerrit.cloudera.org:8080/#/c/20485/3/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20485/3/tests/custom_cluster/test_events_custom_configs.py@403
PS3, Line 403: EventProcessorUtils.wait_for_event_processing(self)
Can we also verify all events are skipped? The metric of "events-skipped" can be checked.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Dec 2023 02:34:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10005/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Dec 2023 19:14:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 11: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10032/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 11:26:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10037/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 19:50:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14615/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 20:01:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9999/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Dec 2023 04:05:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 08:32:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10027/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 23:22:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10032/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 06:57:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10020/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 8
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 08:33:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 11
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 06:57:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14646/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 21:00:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10023/
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 18:18:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20485/4/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/20485/4/tests/custom_cluster/test_events_custom_configs.py@409
PS4, Line 409: assert current_skipped_events - prev_skipped_events >= 24
> This only verifies there are events being skipped but not all the events. T
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 5
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 20:32:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10031/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 10
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 00:21:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 13:44:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Dec 2023 19:30:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20485
to look at the new patch set (#7).
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_events_custom_configs.py
3 files changed, 58 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/7
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 7
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10023/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 9
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2023 13:44:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9999/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 6
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 23:36:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 12
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 19:50:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14607/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 04:22:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/20485 )
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1282
PS1, Line 1282: t
> If we already check getLastSyncedEventId(), wouldn't it be simpler to drop
Exactly for that reason, I'm checking the lastSyncEventId() in the canBeBatched(). CanBeBatched() determines if the event can be batched, if the lastSyncEventId() is greater than the eventId then we can ignore the event without batching.
http://gerrit.cloudera.org:8080/#/c/20485/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1284
PS1, Line 1284: msTbl_ = Preconditions.checkNotNull(insertMessage.getTableObj());
: insertPartition_ = insertMessage.getPtnObj();
: } catch (Exception e) {
: throw new MetastoreNotificationException(debugString("Unable to "
: + "parse insert message"), e);
: }
: }
:
: @Ov
> This seems to be the same as for alter table events - a function like bool
Ack
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 2
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Dec 2023 03:26:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10949: Improve batching logic of partition events
Posted by "Sai Hemanth Gantasala (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, k.venureddy2103@gmail.com, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20485
to look at the new patch set (#3).
Change subject: IMPALA-10949: Improve batching logic of partition events
......................................................................
IMPALA-10949: Improve batching logic of partition
events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
---
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M tests/custom_cluster/test_events_custom_configs.py
2 files changed, 38 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/20485/3
--
To view, visit http://gerrit.cloudera.org:8080/20485
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Gerrit-Change-Number: 20485
Gerrit-PatchSet: 3
Gerrit-Owner: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>