You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/08/22 23:21:00 UTC
[jira] [Commented] (IMPALA-7457) Allow StateStore subscribers to
filter keys by a prefix
[ https://issues.apache.org/jira/browse/IMPALA-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589452#comment-16589452 ]
ASF subversion and git services commented on IMPALA-7457:
---------------------------------------------------------
Commit da01f29d303dca1dbc2be30bc75a72d698a9f4d2 in impala's branch refs/heads/master from [~tlipcon]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=da01f29 ]
IMPALA-7457. statestore: allow filtering by key prefix
This adds the ability for a statestore subscriber to specify a key
prefix which acts as a filter. Only topic entries which match the
specified prefix are transmitted to the subscriber.
This patch makes use of the new feature for a small optimization: the
catalogd subscribes to the catalog topic with a key prefix "!" which we
know doesn't match any actual topic items. This avoids the statestore
having to reflect back the catalog contents to the catalogd, since the
catalogd ignored this info anyway.
A later patch will make use of this to publish lightweight catalog
object version numbers in the same topic as the catalog objects
themselves.
The modification to catalogd's topic subscription is covered by existing
tests. A new specific test is added to verify the filtering mechanism.
Change-Id: I6ddcf3bfaf16bc3cd1ba01100e948ff142a67620
Reviewed-on: http://gerrit.cloudera.org:8080/11253
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
> Allow StateStore subscribers to filter keys by a prefix
> -------------------------------------------------------
>
> Key: IMPALA-7457
> URL: https://issues.apache.org/jira/browse/IMPALA-7457
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Catalog
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Fix For: Impala 3.1.0
>
>
> StateStore subscriptions are currently all-or-nothing. However, in some cases it might be useful to filter a subscription. Two use cases come to mind in the near term:
> 1) currently the catalogd publishes objects into a topic. In order to publish, it has to subscribe to that topic. That means that the statestore then "reflects" all of the info right back to the catalogd causing extra network and memory pressure, even though the catalogd ignores the updates.
> 2) in order to implement cache invalidation on "fetch-on-demand" impalads, we want to publish version numbers of catalog objects without sending the whole object. In order to keep SYNC_DDL working, it's much more convenient to publish the version numbers on the same topic as the full objects. However, the "fetch-on-demand" impalads shouldn't waste bandwidth from the statestore with all of the detailed object info.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org