You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "jakub.martyna.comarch" <ja...@comarch.pl> on 2015/11/16 16:03:48 UTC
Re: Problem with CacheLoadOnlyStoreAdapter
Sorry for reposting but I was not subscribed to mailing list
jakub.martyna.comarch wrote
> Hi,
> We are trying to run ignite cache on 2 nodes with
> CacheLoadOnlyStoreAdapter, but we have encountered problem. After loading
> data ( cache.loadCache(null, (Object[]) null) ) we are checking if all
> data are in cache. Unfortunately some objects are missing. The set of
> missing objects is every time different.
>
> We store many different data types in single cache, all implement common
> interface. As key we are using:
>
> public class SpaceAffinityKey implements Serializable {
> private Long id;
> private String className;
> @AffinityKeyMapped
> private Long routingId;
>
> //equals & hash genereted from id and className
> }
>
> class MyCacheLoader extends CacheLoadOnlyStoreAdapter {
> protected synchronized Iterator<Object[]> inputIterator(Object...
> args) {
> List<Object[]> allObjects = new CopyOnWriteArrayList<>();
>
> for (String clazz: allTypesToLoad) {
> Collection data = loadDataFromDataBase(clazz);
> allObjects.addAll();
> //log loaded data
> }
>
> return allObjects.iterator();
> }
>
> @Override
> protected synchronized IgniteBiTuple<SpaceAffinityKey,
> IdentifiableById> parse(Object[] rec, Object... args) {
> IdentifiableById obj = (IdentifiableById) rec[0];
> long routing = (long) rec[1];
> //log parsed data
> return new IgniteBiTuple<SpaceAffinityKey, IdentifiableById>(
> new SpaceAffinityKey(obj.getId(), obj.getClass().getName(), routing),
> obj);
> }
> }
>
> All data from DB are loaded in inputIterator. In parse some objects are
> missing.
>
> Changing configuration of CacheLoadOnlyStoreAdapter to threads=1
> batchSize=1 queueSize=500000 fixes the problem. But it's slow. Moreover it
> suggests that problem was during precessing batched data from loading
> queues, but ignite had not logged any error.
>
> What can I do to improve load performance?
>
> Regards,
> Jakub Martyna
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-CacheLoadOnlyStoreAdapter-tp1966p1968.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Problem with CacheLoadOnlyStoreAdapter
Posted by Denis Magda <dm...@gridgain.com>.
Hi Jakub,
I tried to reproduce your issue writing an advanced test but failed to
catch any issue related to CacheLoadOnlyStoreAdapter.
In addition, looking at the code of this adapter I don't see any reason
that can lead to the situation when store.parse() method is not called
for every object retrieved from store.inputIterator().
What CacheLoadOnluStoreAdapter parameters do you use that let to catch
the situation when store.parse() is not called for every object?
Please try to enable assertions by passing "-ea" parameter to VM
parameters list. Probably some assertion has raised.
BTW, some comments in regards to your code:
1) You don't need to mark "inputIterator" and "parse" with
"synchronized". These methods are already thread-safe.
2) You don't need to copy content of the collection returned from
"loadDataFromDataBase" to CopyOnWriteArrayList. Just return
collection.iterator() from "inputIterator". The store already works with
the iterator in a thread-safe manner.
Regards,
Denis
On 11/16/2015 6:03 PM, jakub.martyna.comarch wrote:
> Sorry for reposting but I was not subscribed to mailing list
>
>
> jakub.martyna.comarch wrote
>> Hi,
>> We are trying to run ignite cache on 2 nodes with
>> CacheLoadOnlyStoreAdapter, but we have encountered problem. After loading
>> data ( cache.loadCache(null, (Object[]) null) ) we are checking if all
>> data are in cache. Unfortunately some objects are missing. The set of
>> missing objects is every time different.
>>
>> We store many different data types in single cache, all implement common
>> interface. As key we are using:
>>
>> public class SpaceAffinityKey implements Serializable {
>> private Long id;
>> private String className;
>> @AffinityKeyMapped
>> private Long routingId;
>>
>> //equals & hash genereted from id and className
>> }
>>
>> class MyCacheLoader extends CacheLoadOnlyStoreAdapter {
>> protected synchronized Iterator<Object[]> inputIterator(Object...
>> args) {
>> List<Object[]> allObjects = new CopyOnWriteArrayList<>();
>>
>> for (String clazz: allTypesToLoad) {
>> Collection data = loadDataFromDataBase(clazz);
>> allObjects.addAll();
>> //log loaded data
>> }
>>
>> return allObjects.iterator();
>> }
>>
>> @Override
>> protected synchronized IgniteBiTuple<SpaceAffinityKey,
>> IdentifiableById> parse(Object[] rec, Object... args) {
>> IdentifiableById obj = (IdentifiableById) rec[0];
>> long routing = (long) rec[1];
>> //log parsed data
>> return new IgniteBiTuple<SpaceAffinityKey, IdentifiableById>(
>> new SpaceAffinityKey(obj.getId(), obj.getClass().getName(), routing),
>> obj);
>> }
>> }
>>
>> All data from DB are loaded in inputIterator. In parse some objects are
>> missing.
>>
>> Changing configuration of CacheLoadOnlyStoreAdapter to threads=1
>> batchSize=1 queueSize=500000 fixes the problem. But it's slow. Moreover it
>> suggests that problem was during precessing batched data from loading
>> queues, but ignite had not logged any error.
>>
>> What can I do to improve load performance?
>>
>> Regards,
>> Jakub Martyna
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Problem-with-CacheLoadOnlyStoreAdapter-tp1966p1968.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.