[jira] [Commented] (IMPALA-7971) Add support to detect insert events from Impala

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829851#comment-16829851 ] 

ASF subversion and git services commented on IMPALA-7971:
---------------------------------------------------------

Commit 15a33d1baaf4723794e52825a5c4082ff7487507 in impala's branch refs/heads/master from Anurag Mantripragada
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=15a33d1 ]

IMPALA-7971: Add support for insert events in event processor.

This patch adds support for detecting and processing insert events
triggered by impala as well as external engines (eg.Hive).

Inserts from Impala will fire an insert event notification.
Using this event, event-processor will refresh table/partition.
Both insert into and overwrite are supported for tables/partitions.

Known Issues:
1. Inserts into tables from Hive are ignored by the event processor
   as these inserts create an ALTER event first followed by an
   INSERT event. The alter will invalidate table making the refresh
   a no-op. Insert into partitions from hive will create an INSERT
   event first followed by an ALTER event. In this case, there is
   an unnecessary table invalidate after a refresh.
2. Existing self-events logic cannot be used for insert events since
   firing insert event does not allow us to modify table parameters in
   HMS. This means we cannot get the CatalogServiceIdentifiers in insert
   events. Therefore, the event-processor will also refresh the tables
   for which insert operation is performed through Impala.

Testing:
1. Added new custom cluster tests to run different insert commands from
hive and verified new data is available in Impala without invalidate
metadata.

2. Added a test in MetastoreEventsProcessor for testing insert events.

Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52
Reviewed-on: http://gerrit.cloudera.org:8080/12889
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>


> Add support to detect insert events from Impala
> -----------------------------------------------
>
>                 Key: IMPALA-7971
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7971
>             Project: IMPALA
>          Issue Type: Sub-task
>            Reporter: Vihang Karajgaonkar
>            Assignee: Anurag Mantripragada
>            Priority: Major
>
> When data is inserted into existing tables and partitions, Catalog does not issue any metastore API calls. Metastore provides a API called {{fire_listener_event}} which can be used to add a {{INSERT_EVENT}} to the metastore notification log. This event can be used by other Impala instances to invalidate or update the filemetada information when data is inserted or overrwriten on a given table or partition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org