You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/12/15 04:35:00 UTC
[jira] [Commented] (IMPALA-10949) Improve batching logic of events
[ https://issues.apache.org/jira/browse/IMPALA-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797003#comment-17797003 ]
ASF subversion and git services commented on IMPALA-10949:
----------------------------------------------------------
Commit a6de494f24c47fbd679a037341ae0a34b9f696ff in impala's branch refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a6de494f2 ]
IMPALA-10949: Improve batching logic of partition events
Currently the batching logic for partitions evaluates self-event check
by looking at the event id of the last event in the batch. This patch
improves the batching logic by evaluating table's lastSyncEventId to
the current event's event id and decide whether to batch the event or
not. This way we can have fewer batch sizes and avoid self events if
any, beforehand.
Testing: Added an end-to-end test to verfiy the batch size for
ALTER_PARTITION batch events.
Change-Id: I4e79510739347cbe669719a9e4cb9cabd5daa7d3
Reviewed-on: http://gerrit.cloudera.org:8080/20485
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Improve batching logic of events
> --------------------------------
>
> Key: IMPALA-10949
> URL: https://issues.apache.org/jira/browse/IMPALA-10949
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Vihang Karajgaonkar
> Assignee: Sai Hemanth Gantasala
> Priority: Major
>
> This is a followup based on the review comment https://gerrit.cloudera.org/#/c/17848/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1641
> Current approach of batching batches together the events from a single operation so that self-event check is done per-batch. However, it looks like there is a considerable scope of improving the batching logic by clubbing together accross the various sources of the events on a table when IMPALA-10926 is merged. After IMPALA-10926 each table will track the last_synced_event and then the events processor can simply ignore a event which <= the last_synced_event. This simplification of self-events logic will enable easier batching for all the events of a type on a table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org