[jira] [Commented] (IMPALA-11028) Table loading could fail if metastore cleans up old events

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448395#comment-17448395 ] 

ASF subversion and git services commented on IMPALA-11028:
----------------------------------------------------------

Commit cc6f6d5c91ba1db3fca83c65f7d2f87c98077025 in impala's branch refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=cc6f6d5 ]

IMPALA-11028: Table loading can fail when events are cleaned up

IMPALA-10502 introduces a createEventId field of a table which
is updated when Impala creates a table. This is used by
the events processor to determine if the subsequent CREATE_TABLE
event which is received should be skipped or not.

When the table is loaded for the first time, in order to avoid
race conditions, TableLoader updates the createEventId to the
last CREATE_TABLE event id from the metastore. In order to
fetch the latest CREATE_TABLE event id, it fetches all the
events from metastore since the last known createEventId of the
table. However, if there is a significant delay between
(more than 24hrs) between the time a table is created
or invalidated, and the table is queried, it is possible that
the metastore cleanup thread deletes the events which are generated
since the table's createEventId. In such a case, the HMS Client method
getNextNotification() throws an IllegalStateException due to the missing
events. This exception causes the Table load to fail and query to error
out.

The fix is to not rely on the HMS Client method which throws the
IllegalStateException. Instead we use the backing thrift API directly.

Testing:
1. Introduced a custom cluster test which can reproduce this issue.
2. Test works after the patch.
3. Core tests.

Change-Id: I95e5e20e1a2086688a92abdfb28e89177e996a1a
Reviewed-on: http://gerrit.cloudera.org:8080/18038
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Table loading could fail if metastore cleans up old events
> ----------------------------------------------------------
>
>                 Key: IMPALA-11028
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11028
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>            Priority: Major
>
> After IMPALA-10502, Catalogd tracks the table's create event id. When the table is loaded for the first time, it updates the create event id of the table. But if the table is loaded for the first time after a long delay (after 24 hrs) it is possible the metastore cleans up old notification logs entries which are required by catalogd during the table load.
> See this snippet from TableLoader.java
> {noformat}
>       if (eventId != -1 && catalog_.isEventProcessingActive()) {
>         // If the eventId is not -1 it means this table was likely created by Impala.
>         // However, since the load operation of the table can happen much later, it is
>         // possible that the table was recreated outside Impala and hence the eventId
>         // which is stored in the loaded table needs to be updated to the latest.
>         // we are only interested in fetching the events if we have a valid eventId
>         // for a table. For tables where eventId is unknown are not created by
>         // this catalogd and hence the self-event detection logic does not apply.
>         events = MetastoreEventsProcessor.getNextMetastoreEvents(catalog_, eventId,
>             notificationEvent -> CreateTableEvent.CREATE_TABLE_EVENT_TYPE
>                 .equals(notificationEvent.getEventType())
>                 && notificationEvent.getDbName().equalsIgnoreCase(db.getName())
>                 && notificationEvent.getTableName().equalsIgnoreCase(tblName));
>       }
> {noformat}
> {{getNextMetastoreEvents}} method can throw the following exception if the metastore has cleaned up older entries (by default 24 hrs). This is controlled by configuration {{hive.metastore.event.db.listener.timetolive}} on the metastore side.
> I could reproduce the problem setting the following metastore configs.
> {noformat}
> hive.metastore.event.db.listener.clean.interval=10s
> hive.metastore.event.db.listener.timetolive=120s
> {noformat}
> Now run the following Impala script
> {noformat}
> create table t1 (c1 int);
> create table t2 (c1 int);
> select sleep(240000);
> create table t3 (c1 int);
> select * from t1;
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org