[jira] [Assigned] (IMPALA-12933) Catalogd should set eventTypeSkipList when fetching specifit events for a table

["Sai Hemanth Gantasala (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sai Hemanth Gantasala reassigned IMPALA-12933:
----------------------------------------------

    Assignee: Quanlong Huang  (was: Sai Hemanth Gantasala)

> Catalogd should set eventTypeSkipList when fetching specifit events for a table
> -------------------------------------------------------------------------------
>
>                 Key: IMPALA-12933
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12933
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>
> There are several places that catalogd will fetch all events of a specifit type on a table. E.g. in TableLoader#load(), if the table has an old createEventId, catalogd will fetch all CREATE_TABLE events after that createEventId on the table.
> Fetching the list of events is expensive since the filtering is done on client side, i.e. catalogd fetch all events and filter them locally based on the event type and table name:
> [https://github.com/apache/impala/blob/148888e3ed4f97292499b2e6ee8d5a756dc648d9/fe/src/main/java/org/apache/impala/catalog/TableLoader.java#L98-L102]
> [https://github.com/apache/impala/blob/b7ddbcad0dd6accb559a3f391a897a8c442d1728/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L336]
> This could take hours if there are lots of events (e.g 1M) in HMS. In fact, NotificationEventRequest can specify an eventTypeSkipList. Catalogd can do the filtering of event type in HMS side. On higher Hive versions that have HIVE-27499, catalogd can also specify the table name in the request (IMPALA-12607).
> This Jira focus on specifying the eventTypeSkipList when fetching events of a particular type on a table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org