ScanQuery log all entries

["javastuff.sam@gmail.com" <ja...@gmail.com>]

Hi,

I have a usecase where I have to log cached entries. I am doing below -

/ScanQuery<String, Object> scan = new ScanQuery<>();
cache.query(scan).getAll();/

Question -
1. With OFF_HEAP cache it is suppose to be huge number of entries in cache.
Does getAll() will try to bring all into heap and potential ou-of-memory?
2. Is there other way to iterate page by page?
3. Is there any other restriction I can add to scan query like limit 1000 or
query timeout?

Thanks,
-Sam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery log all entries

[dkarachentsev <dk...@gridgain.com>]

Sam,

Cursor iterator will go over whole result, but it will not hold in memory
all dataset, just part limited by page size. In other words, it fetches 1024
entries, and when iterator points after last entry - loads another page with
1024 entries. So you shouldn't worry when results arrived - they
automatically paginated, you just need correctly manage data, e.g. if you
collect all of them in list - you may get OOME.

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970p12065.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery log all entries

["javastuff.sam@gmail.com" <ja...@gmail.com>]

Thanks. One question though -

Below loop will automatically go over all pages or just one page with 1024
entries?
       
            for (Cache.Entry<AffinityKey&lt;Long>, Person> entry : cursor) 
                System.out.println("Key: " + entry.getKey() + ", Value: " +
entry.getValue()); 

If it is just one page then how to know there is more results and how to
iterate on those?

Thanks,
-Sam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970p12047.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery log all entries

[dkarachentsev <dk...@gridgain.com>]

Hi,

Yes, you understand correctly. Here is example:

private static void sqlQuery() {
        IgniteCache<AffinityKey&lt;Long>, Person> cache =
Ignition.ignite().cache(PERSON_CACHE);

        // SQL clause which selects salaries based on range.
        String sql = "salary > ? and salary <= ?";

        // People with salaries between 1000 and 2000.
        try (QueryCursor<Cache.Entry&lt;AffinityKey&lt;Long>, Person>>
cursor =
                 cache.query(new SqlQuery<AffinityKey&lt;Long>,
Person>(Person.class, sql).
                 setArgs(1000, 2000))) {

            // Cursor is Iterable, so you may iterate over it with for each
loop.
            for (Cache.Entry<AffinityKey&lt;Long>, Person> entry : cursor)
                System.out.println("Key: " + entry.getKey() + ", Value: " +
entry.getValue());
        }
}

It's based on
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970p12017.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery log all entries

["javastuff.sam@gmail.com" <ja...@gmail.com>]

Thanks Dmitry.

So it means if I use QueryCursor, it will not bring all entries in heap but
only 1024 or page size setting.
Is there any example for QueryCursor? Can anybody share a sample code
snippet for QueryCursor usage?

Thanks,
-Sam 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970p12003.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: ScanQuery log all entries

[dkarachentsev <dk...@gridgain.com>]

Hi Sam,

1. Yes getAll() will bring all values to an on-heap collection, which may
lead to OOME, so it should be used with caution.
2. Any query returns QueryCursor that lazily iterates over entries. You may
do 1000 iterations over cursor and accumulate result in some collection.
This will be just like pagination.
3. You may set page size, by default it's 1024. This means that cursor will
pre-fetch from remote node 1024 entries.

-Dmitry



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ScanQuery-log-all-entries-tp11970p11981.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.