You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/04/09 05:00:00 UTC
[jira] [Commented] (IMPALA-8632) Add support for self-event
detection for insert events
[ https://issues.apache.org/jira/browse/IMPALA-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078919#comment-17078919 ]
ASF subversion and git services commented on IMPALA-8632:
---------------------------------------------------------
Commit e0ed7d321c1dca85a0f1482842f8db6db517c909 in impala's branch refs/heads/master from xiaomeng
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e0ed7d3 ]
IMPALA-8632: Add support for self-event detection for insert events
In case of INSERT_EVENTS if Impala inserts into a table it causes a
refresh to the underlying table/partition. This could be unnecessary
when there is only one Impala cluster in the system.
We can detect a self-event in such cases when the HMS API to fire a
listener event returns the event id. This is used by EventProcessor
to ignore the event when it is fetched later in the next polling cycle.
Testing:
Add testInsertFromImpala() in MetastoreEventsProcessorTest.java to test
insert event self-event detection when insert into table and partition.
Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf
Reviewed-on: http://gerrit.cloudera.org:8080/15648
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Add support for self-event detection for insert events
> ------------------------------------------------------
>
> Key: IMPALA-8632
> URL: https://issues.apache.org/jira/browse/IMPALA-8632
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Vihang Karajgaonkar
> Assignee: Xiaomeng Zhang
> Priority: Critical
>
> In case of {{INSERT_EVENTS}} if Impala inserts into a table it causes a refresh to the underlying table/partition. This could be unnecessary when there is only one Impala cluster in the system. The existing self-event detection framework cannot identify such events because they are not sending HMS objects like tables and partitions to the HMS. Instead in case of {{INSERT_EVENT}} HMS API only asks for a table name or partition value to fire a insert event on it.
> We can detect a self-event in such cases if the HMS API to fire a listener event is improved to return the event id. This would be used by EventProcessor to ignore the event when it is fetched later in the next polling cycle. In order to support this, we will need to make a change to Hive as well so that the enhanced API can be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org