Continuous query setPageSize and setTimeInterval question

[crypto_ricklee <ri...@crypto.com>]

Dear all,

I tried to play around with the setPageSize and setTimeInterval on
Continuous Query, but it doesn't seem to have any effect. My code is like:

  private void configOrderListener(Ignite ignite) {
    IgniteCache<OrderPK, Order> cache =
ignite.cache(StoreHelper.getCacheName(OrderStore.class));

    ContinuousQuery<OrderPK, Order> qry = new ContinuousQuery<>();

//    qry.setPageSize(20);
    qry.setLocal(true);
    qry.setTimeInterval(2000L);
    qry.setLocalListener(createOrderHandler());

    cache.query(qry);
  }

  private CacheEntryUpdatedListener<OrderPK, Order> createOrderHandler() {
    return (evts) -> {
      log.info("batch size: {}", Iterables.size(evts));

      evts.forEach(
              e -> {
                if (e.getEventType().equals(EventType.CREATED) ||
e.getEventType().equals(EventType.UPDATED)) {
                  log.info("{}", e);
                  redisService.publish("user.order." +
e.getValue().getSymbol() + "." + e.getValue().getKey().getAccountUuid(),
e.getValue()).block();
                }
              });
    };
  }


What I expect is I will be receiving event every 2000ms, but the result was
I received each event immediately without any batching. I've tried to set
the page size, but no effect as well. Am I doing sth wrong? Please kindly
advise. 

Thanks,
Rick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

[akorensh <al...@gmail.com>]

It is likely entries are being quickly produced and filling up the buffer
giving the effect of an immediate update. 
You can test it via specific delays and logging.
Make the server print out the counter of the object.
say:
  if the key is an integer and the value is too then print out the key as
you put it in.
  do the same on the client when it receives the object.

Produce a number less than the buffer size , put in a 10 second delay, see
what the client gives then produce more items, and again observe the client. 

You can also set pageSize to a very large number -- say 40000 and watch it
update in intervals. -- be aware that here memory effects might come into
play especially if your objects are large.


Here is an example:
  the server will produce 100 then sleep.
  the client has set the pageCount to 1000
  for every 10 "sleeping.." it will print out the 1000 entries. 

*server:*
           int i = 0;
        int sleepCounter = 1;
        while (true) {
            cache.put(i++, Integer.toString(i));
            System.out.println("added entry: " + i);
            if(i%100 == 0){
                System.out.println("sleeping: " + sleepCounter++);
                if(sleepCounter %10 == 0) sleepCounter = 0;
                Thread.sleep(1000);
            }
        }

*client: *
         ContinuousQuery < Integer, String > qry = new ContinuousQuery<>();
         qry.setTimeInterval(0);
         qry.setPageSize(1000);
         qry.setLocalListener((evts) -> evts.forEach(e ->
System.out.println("key=" + e.getKey() + ", val=" + e.getValue())));
          cache.query(qry);






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

[crypto_ricklee <ri...@crypto.com>]

This was what I thought, however, no matter what number I set to the
pageSize, e.g., 5, 20, 100, my local listener got the update immediately, 1
by 1, but not batching by the page size...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

[akorensh <al...@gmail.com>]

You are correct. Your local reciever will get an update every 100 entries.

setPageSize sets the number of entries to batch together before sending.
When the server has accumulated a number of entries larger than getPageSize
it sends a message to the receiver. 

Like I mentioned before, setIntervalSize() allows you to send a batch every
set interval irrespective of whether pageSize has been reached or not

from the doc:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/ContinuousQuery.html

Continuous queries allow registering a remote filter and a local listener
for cache updates. 
If an update event passes the filter, it will be sent to the node that
executed the query, and local listener will be notified.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

[crypto_ricklee <ri...@crypto.com>]

Thanks Alex,

But I still not quite understand the expected behaviour. If I set the page
size to 100 and interval to 0, should the local query be triggered for every
100 updates?

Regards,
Rick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query setPageSize and setTimeInterval question

[akorensh <al...@gmail.com>]

Hi,
   setTimeInterval limits the time Ignite will wait for an internal buffer
to fill up before sending.

   It is not a time delay setting.

   From the doc:
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/ContinuousQuery.html#setTimeInterval-long-
     When a cache update happens, entry is first put into a buffer. Entries
from buffer will be sent to the 
     master node only if the buffer is full (its size can be provided via
Query.setPageSize(int) method) or time 
     provided via this method is exceeded.

     Default time interval is 0 which means that time check is disabled and
entries will be sent only when 
    buffer is full.
   
Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/