getAsMap ??

[Chris Berry <ch...@gmail.com>]

Hi,

I need to get a copy of a REPLICATED cache – as a ImmutableMap that I can
exercise some mapping functions against.

Something like this:

public List<Foo> findMatches(SearchTerms searchTerms) {
    ImmutableMap<String, Foo> cache = cache.getAsMap();
    return matchingStrategy.match(cache, searchTerms);
}

public class MatchingStrategy {
    public List<Foo> match(Map<String, Foo> cache, SearchTerms searchTerms)
{
        return cache.values().stream()
            .filter(contract -> searchTerms.matches(contract.getKey()))
            .collect(Collectors.toList());
    }
}

where I’m looking for a IgniteCache::getAsMap() 
Or some way to implement that.

But I’m sure there is a much better way to do this??
That cannot be performant.

NOTE: this cache is relatively small and slow changing.

Any help is greatly appreciated.

Thanks, 
-- Chris 




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/getAsMap-tp16242.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: getAsMap ??

[dkarachentsev <dk...@gridgain.com>]

Hi Chris,

1. Indexing may slowdown insertions, but in will not be used if no
CacheConfiguration.setIndexedTypes() or
CacheConfiguration.setQueryEntities() set.

2. It depends on conditions. If you have a lot of data, but you need to
filter out a small set of them, then indexing may greatly help you, because
it will use index but full scan. In other hand, scan query may be greatly
optimized if it's used locally and per partition, but again, it will check
all entries.

3. If you set withKeepBinary(), then you'll be working with BinaryObject and
it's not require any specific config.

Thanks!
-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/getAsMap-tp16242p16347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: getAsMap ??

[Chris Berry <ch...@gmail.com>]

Thank you for the speedy reply.
I found that same information when I dug deeper last night.

And after working thru the annoying IncompatibleClassChangeError issue and
mismatched H2 classes,
I got it working.

For posterity, I ended up with this:

    public Set<TKey> getCacheKeys() {
        String tableName = valueClass.getSimpleName();
        SqlFieldsQuery qry = new SqlFieldsQuery("select _key from " +
tableName);
        Collection<List&lt;?>> res =  igniteCacheHandle.query(qry).getAll();

        Set<TKey> keys = new HashSet<>();
        for(List<?> row : res) {
            keys.add((TKey)row.get(0));
        }
        return keys;
    }

Using a SqlFieldsQuery.

And I use it like this:

    public List<Foo > findMatches(SearchTerms searchTerms) {
        Map<String, Foot> map = cache.getAll(cache.getCacheKeys());
        return matchingStrategy.match(map, searchTerms);
    }

Which is pretty inefficient I think. But I wanted to get something working….

I have a few questions.

1) I have only one REPLICATED cache – out of many caches -- that I need to
do a Query against.
The rest are all used as simple KV stores.
Have I introduced an inefficiency by introducing the ignite-indexing
module??
Is there a way to use it ONLY for 1 cache??
Or is indexing always present, and I have just now become aware of it??
(This seems unlikely since the H2 DB issue should have bitten me sooner??)

2) Would it be more efficient to use a ScanQuery??
Or is the SqlFieldsQuery roughly equivalent??

NOTE: I assume to do this whole thing more efficiently I would use a
ScanQuery with a Filter??
Which would eliminate the need for the interim copies.

3) I am using:  igniteConfig.setMarshaller(new BinaryMarshaller())  --
paired with:   cache.withKeepBinary()
Do I need to do anything special in this case??
Presently it seems not?? 
I am using only this: igniteCacheConfig.setIndexedTypes(keyClass,
valueClass);
And it all is working as expected.
Still. If I used IgniteBiPredictae would it be more efficient ??

Thanks agaian for your help.

Cheers, 
-- Chris 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/getAsMap-tp16242p16259.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: getAsMap ??

[dkarachentsev <dk...@gridgain.com>]

Hi Chris,

Collecting all data from cache to map is a bad idea, because it could be
quite large and you may get OutOfMemoryError. Exactly for that purpose
ScanQuery was designed [1]. You may set filter and using cursor get data:
iterating over it (this will load entries from grid with batches), or call
getAll() [2]. Filtering will be processed on data node locally with maximal
performance.

[1] https://apacheignite.readme.io/v1.1/docs/cache-queries#scan-queries
[2]
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheQueryExample.java

Thanks!



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/getAsMap-tp16242p16255.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.