You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by zork <pr...@gmail.com> on 2020/05/17 19:14:37 UTC
Continuous Query on a varying set of keys
Hi,
We have a table in ignite cache which would have say around 1Mn entries at
anytime. Now we wish to listen on updates on a subset of these keys (say
5-10 thousand keys) and this subset keeps on changing as the user
subscribes/unsubscribes to these keys.
The way it is currently working is one continuous query is attached for
every key whenever it is subscribed and it is closed whenever that key is no
longer of interest (or unsubscribed). The problem with this is that since
there are so many continuous queries (a few thousands), the application goes
out of memory. Also, it would mean all those queries would be evaluated on
the remote node for every update.
To overcome this, what we intend to do is to have just one continuous query
which would listen to all the updates on this table (i.e. all the keys) and
on receiving these updates we would have to filter those of our interest on
our end. But this would mean unnecessary updates would flow over the network
and it doesn't sound like a very good solution too.
Can someone suggest a better way this problem could be addressed? Do we have
something else in ignite to cater such requirement?
Thanks in advance.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous Query on a varying set of keys
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Yes, using other continuous query to watch changes of this set looks OK. Of
course there will be some yak shaving to do.
Regards,
--
Ilya Kasnacheev
вт, 26 мая 2020 г. в 12:28, zork <pr...@gmail.com>:
> Thanks.
> So I can think of two ways using which such a set could be maintained by
> the
> remote node:
> 1. The remote node listens to a new topic through which the local node
> sends
> it a message whenever the set changes.
> 2. Or, the local node puts the set values in a new table in the cache
> itself
> and remote node can maybe listen to it using another continuous query.
> Is that right?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Continuous Query on a varying set of keys
Posted by zork <pr...@gmail.com>.
Thanks.
So I can think of two ways using which such a set could be maintained by the
remote node:
1. The remote node listens to a new topic through which the local node sends
it a message whenever the set changes.
2. Or, the local node puts the set values in a new table in the cache itself
and remote node can maybe listen to it using another continuous query.
Is that right?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous Query on a varying set of keys
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
No, updating local set will not be sent over to the remote node, but on the
remote node you can probably update it from your filter:
qry.setRemoteFilterFactory(new Factory<CacheEntryEventFilter<Integer,
String>>() {
@Override public CacheEntryEventFilter<Integer, String>
create() {
return new CacheEntryEventFilter<Integer, String>()
{
@Override public boolean
evaluate(CacheEntryEvent<? extends Integer, ? extends String> e) {
// This method is implemented and called
on remote node
set = updateSetIfNeeded(set);
return set.contains(e.getKey());
}
};
}
});
Please note that it is not advisable to do cache operations from filter, so
you should probably do that in background by e.g. registering a service.
Regards,
--
Ilya Kasnacheev
вт, 26 мая 2020 г. в 11:31, zork <pr...@gmail.com>:
> Hi,
> Sorry but I could not get it to work.
>
> The standard way of defining a remote filter as shown in sample repo is
> something like:
>
> qry.setRemoteFilterFactory(new Factory<CacheEntryEventFilter<Integer,
> String>>() {
> @Override public CacheEntryEventFilter<Integer, String>
> create() {
> return new CacheEntryEventFilter<Integer, String>()
> {
> @Override public boolean
> evaluate(CacheEntryEvent<? extends Integer, ? extends String> e) {
> return e.getKey() > 10;
> }
> };
> }
> });
>
> In the above, instead of having *10* constant, I need to have a HashSet
> from
> which I can check if the updated key exists in it or not (see the snippet
> below) And I need the changes in the HashSet to be reflected in the filter.
> However it's not making sense to me because the HashSet which is modified
> is
> on one node while the filter is on another node so I expect the remote node
> would already have it serialized when the filter was first created and it
> would not change even if the set changes in the local node.
>
> HashSet<Integer> = new HashSet<>();
> set.add(20);
> qry.setRemoteFilterFactory(new Factory<CacheEntryEventFilter<Integer,
> String>>() {
> @Override public CacheEntryEventFilter<Integer, String>
> create() {
> return new CacheEntryEventFilter<Integer, String>()
> {
> @Override public boolean
> evaluate(CacheEntryEvent<? extends Integer, ? extends String> e) {
> return set.contains(e.getKey());
> }
> };
> }
> });
> set.add(10) // would this actually change the filter on remote node?
>
> Perhaps I'm missing something very obvious here. Please help me identify
> it.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Continuous Query on a varying set of keys
Posted by zork <pr...@gmail.com>.
Hi,
Sorry but I could not get it to work.
The standard way of defining a remote filter as shown in sample repo is
something like:
qry.setRemoteFilterFactory(new Factory<CacheEntryEventFilter<Integer,
String>>() {
@Override public CacheEntryEventFilter<Integer, String>
create() {
return new CacheEntryEventFilter<Integer, String>()
{
@Override public boolean
evaluate(CacheEntryEvent<? extends Integer, ? extends String> e) {
return e.getKey() > 10;
}
};
}
});
In the above, instead of having *10* constant, I need to have a HashSet from
which I can check if the updated key exists in it or not (see the snippet
below) And I need the changes in the HashSet to be reflected in the filter.
However it's not making sense to me because the HashSet which is modified is
on one node while the filter is on another node so I expect the remote node
would already have it serialized when the filter was first created and it
would not change even if the set changes in the local node.
HashSet<Integer> = new HashSet<>();
set.add(20);
qry.setRemoteFilterFactory(new Factory<CacheEntryEventFilter<Integer,
String>>() {
@Override public CacheEntryEventFilter<Integer, String>
create() {
return new CacheEntryEventFilter<Integer, String>()
{
@Override public boolean
evaluate(CacheEntryEvent<? extends Integer, ? extends String> e) {
return set.contains(e.getKey());
}
};
}
});
set.add(10) // would this actually change the filter on remote node?
Perhaps I'm missing something very obvious here. Please help me identify it.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous Query on a varying set of keys
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Remote filter is code. It can execute arbitrary logic. It can adjust to
what it needs to filter, change its own behavior with time.
Regards,
--
Ilya Kasnacheev
пн, 18 мая 2020 г. в 15:40, zork <pr...@gmail.com>:
> Hi Ilya,
> Thanks for your response.
> I'm aware of remote filters but can these filters be modified once the
> query
> is already attached?
> Because if not, then this would not solve my use case as the filter would
> always give me updates on a fixed subset of keys, however in my case this
> subset is varying (based on what keys a user subscribes from the GUI).
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Continuous Query on a varying set of keys
Posted by zork <pr...@gmail.com>.
Hi Ilya,
Thanks for your response.
I'm aware of remote filters but can these filters be modified once the query
is already attached?
Because if not, then this would not solve my use case as the filter would
always give me updates on a fixed subset of keys, however in my case this
subset is varying (based on what keys a user subscribes from the GUI).
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous Query on a varying set of keys
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Continuous query has a notion of 'remote filter'. This is a piece of code
which is executed near data (on server nodes) to determine if the update
needs to be sent over network.
https://apacheignite.readme.io/docs/continuous-queries#remote-filter
If you define a proper remote filter, updates will not flow over the
network unless this is actually needed.
Regards,
--
Ilya Kasnacheev
вс, 17 мая 2020 г. в 22:14, zork <pr...@gmail.com>:
> Hi,
>
> We have a table in ignite cache which would have say around 1Mn entries at
> anytime. Now we wish to listen on updates on a subset of these keys (say
> 5-10 thousand keys) and this subset keeps on changing as the user
> subscribes/unsubscribes to these keys.
>
> The way it is currently working is one continuous query is attached for
> every key whenever it is subscribed and it is closed whenever that key is
> no
> longer of interest (or unsubscribed). The problem with this is that since
> there are so many continuous queries (a few thousands), the application
> goes
> out of memory. Also, it would mean all those queries would be evaluated on
> the remote node for every update.
>
> To overcome this, what we intend to do is to have just one continuous query
> which would listen to all the updates on this table (i.e. all the keys) and
> on receiving these updates we would have to filter those of our interest on
> our end. But this would mean unnecessary updates would flow over the
> network
> and it doesn't sound like a very good solution too.
>
> Can someone suggest a better way this problem could be addressed? Do we
> have
> something else in ignite to cater such requirement?
>
> Thanks in advance.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>