You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/03/09 14:14:00 UTC
[jira] [Commented] (IMPALA-11822) Optimize the Refresh/Invalidate event processing by skipping unnecessary events
[ https://issues.apache.org/jira/browse/IMPALA-11822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698406#comment-17698406 ]
ASF subversion and git services commented on IMPALA-11822:
----------------------------------------------------------
Commit 148888e3ed4f97292499b2e6ee8d5a756dc648d9 in impala's branch refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=148888e3e ]
IMPALA-11822: Optimize the Refresh/Invalidate event processing by skipping unnecessary events
Added a new variable 'lastRefreshEventId' in the
catalogD's table/partition object to store the
latest event id before loading the table/partition.
This will be updated frequently based on refresh
or invalidate commands. This variable can be used
in the event processor to decide whether to process
or skip the reload event by comparing it with the
current event id. It is enough to store the refresh
event's event id, invalidate event anyway flushes
out the object from cache.
Note: Need to enable two configs for this
optimization to work:
1) enable_reload_events=true
2) enable_sync_to_latest_event_on_ddls=true
Testing: Added a test to fire few reload events via
HMS API and then verify in the event processor that
some older events are skipped.
Change-Id: I905957683a96c3ea01ab4bf043d6658ce37b7574
Reviewed-on: http://gerrit.cloudera.org:8080/19484
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Optimize the Refresh/Invalidate event processing by skipping unnecessary events
> -------------------------------------------------------------------------------
>
> Key: IMPALA-11822
> URL: https://issues.apache.org/jira/browse/IMPALA-11822
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Sai Hemanth Gantasala
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
>
> Optimize the Refresh/Invalidate event processing by skipping unnecessary events.
> Currently, we process every event as a new event. Consider there are 5 refresh events of the same table in the event processor queue. We can process the first refresh event and skip the remaining 4 events by comparing the timestamp of the refreshed table and event time. This way we can greatly improve the performance by skipping unnecessary events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org