Issue with large number of Continuous Queries

[Dorah <pr...@iongroup.com>]

*Topology*:

Server-1 --> Cache myCache, holds continuously updating data like market
data(prices, status, tradetime etc) for instruments. InstrumentId is the key
for this cache.
                   Server-1 running with following jvm params: -Xms1g,-Xmx6g

Client-1 --> Pushing continuous updates on the cache

Client - 2 & 3 --> Listening updates on myCache using separate Continuous
query on every key (i.e. one continuous query per instrumentId).

The Cache Configuration is as follows:
          cacheMode    PARTITIONED
          atomicityMode    ATOMIC
          backups    2
          readFromBackup    true
          copyOnRead    true
          statisticsEnabled    true
          managementEnabled    true

System hardware: 8 core, 32gb RAM
For now all servers and client below run on same machine.

---------------------------------------------------------

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when high number continuous queries machine reaches 100% CPU
utilization.

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

<http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled.png> 

But now the problem is that both the clients do not need to listen to
updates in all the keys in cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

<http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled2.png> 

Would this be the correct way to achieve what we are trying to achieve? Or
could this be done some other better way in ignite?

Follow up Question:
How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issue with large number of Continuous Queries

[Andrei Aleksandrov <ae...@gmail.com>]

Hi,

Please read some comments below:

1)You said:

*Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
updates per second on every item. Lets say both client-2 and client-3 have
1000 different continuous queries open to listen to every update.

Continues query can handle all updates for some cache. You shouldn't 
start special CQ for every entry. Please read the following:

https://apacheignite.readme.io/docs/continuous-queries

2)You said:

With the above scenario, we are observing the server-1 alone taking 60-70%
CPU and about 4GB memory.
In this case when a high number of continuous queries machine reaches 
100% CPU
utilization.

You can try to monitor what Ignite parts take the CPU and memory usage. 
Taking into account the point 1) - you shouldn't have a lot of CQ for 
the same cache.

3)You said:

*Thinking to fix as*: Use single continuous query per client to listen to
all the updates. i.e. there would be just one continuous query and it would
listen to all the updates.

This is the correct way.

4)You said:

But now the problem is that both the clients do not need to listen to
updates in all the keys in the cache. So we are thinking of adding another
column to the ignite cache using which we can filter the updates by checking
if the client column of the updated row contains the client name for which
filter is being checked. e.g. the new table would look like-

Correct you can filter the updates using the following:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/AbstractContinuousQuery.html#getRemoteFilterFactory--

5)You said:

How many continuous queries can ignite handle at once with the configuration
we mentioned or is there any such benchmark values available on any
particular configuration? Is it fine to have as many as 1000 (or even more)
continuous queries running at once? If yes, how can we make it less CPU
intensive and more performant?

I don't think that the current information can be provided. It depends 
on many things and should be tested in your specific environment. 
However, I suggest to don't set so more QE without reason.

BR,
Andrei

2/11/2020 9:42 AM, Dorah пишет:
> *Topology*:
>
> Server-1 --> Cache myCache, holds continuously updating data like market
> data(prices, status, tradetime etc) for instruments. InstrumentId is the key
> for this cache.
>                     Server-1 running with following jvm params: -Xms1g,-Xmx6g
>
> Client-1 --> Pushing continuous updates on the cache
>
> Client - 2 & 3 --> Listening updates on myCache using separate Continuous
> query on every key (i.e. one continuous query per instrumentId).
>
> The Cache Configuration is as follows:
>            cacheMode    PARTITIONED
>            atomicityMode    ATOMIC
>            backups    2
>            readFromBackup    true
>            copyOnRead    true
>            statisticsEnabled    true
>            managementEnabled    true
>
> System hardware: 8 core, 32gb RAM
> For now all servers and client below run on same machine.
>
> ---------------------------------------------------------
>
> *Scenario*: There are 1000 items in the myCache and client-1 is pushing 3
> updates per second on every item. Lets say both client-2 and client-3 have
> 1000 different continuous queries open to listen to every update.
>
> With the above scenario, we are observing the server-1 alone taking 60-70%
> CPU and about 4GB memory.
> In this case when high number continuous queries machine reaches 100% CPU
> utilization.
>
> *Thinking to fix as*: Use single continuous query per client to listen to
> all the updates. i.e. there would be just one continuous query and it would
> listen to all the updates.
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled.png>
>
> But now the problem is that both the clients do not need to listen to
> updates in all the keys in cache. So we are thinking of adding another
> column to the ignite cache using which we can filter the updates by checking
> if the client column of the updated row contains the client name for which
> filter is being checked. e.g. the new table would look like-
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2776/Untitled2.png>
>
> Would this be the correct way to achieve what we are trying to achieve? Or
> could this be done some other better way in ignite?
>
> Follow up Question:
> How many continuous queries can ignite handle at once with the configuration
> we mentioned or is there any such benchmark values available on any
> particular configuration? Is it fine to have as many as 1000 (or even more)
> continuous queries running at once? If yes, how can we make it less CPU
> intensive and more performant?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/