You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by diopek <de...@gmail.com> on 2017/03/01 14:15:24 UTC
Re: Missing records Ignite cache size grows
Below is our implementation of LoadOnlyCacheStore inputIterator method.
As we tested many times there is no issue on uniqueness of generated cache
keys.
But still after loading high number of records into IgniteCache, records
count doesn't match, and few 10s of keys were missing out of million
records. Also if we use the Java cache populated just before serializing
this cache into IgniteCache, all number of records are matching.
In short, after several days of try-outs and debugging, we narrowed down the
root cause of issue to LoadOnlyCacheStore base class. So, at this point, we
need Ignite core team support to resolve this bug.
Thanks
Yes, we checked uniqueness of key several times. Below is representative
snippet of our code, how we generate cache key inside inputIterator method.
@Override
protected Iterator<Tuple2<Long,ArrayList<MyDTO>>>*
inputIterator*(Object... args) throws CacheLoaderException {
Iterator<Tuple2<Long,ArrayList<MyDTO>>> iterator = null;
ResultSetExtractor<ArrayList<Tuple2<Long,ArrayList<MyDTO>>>>
extOrRepMapResultSetExtractor = new
ResultSetExtractor<ArrayList<Tuple2<Long,ArrayList<MyDTO>>>>() {
@Override
public ArrayList<Tuple2<Long,ArrayList<MyDTO>>>
extractData(ResultSet rs)
throws SQLException, DataAccessException {
ArrayList<Tuple2<Long,ArrayList<MyDTO>>>
extOrRepList = new
ArrayList<Tuple2<Long,ArrayList<MyDTO>>>(startSize);
*final AtomicLong entryCnt = new AtomicLong(0);*
while (rs.next()) {
extOrRepList.add(new Tuple2<Long,
ArrayList<MyDTO>>(*entryCnt.incrementAndGet()*,prevDTOList));
}
return extOrRepList;
}
};
jdbcTemplate.setFetchSize(SQL_FETCH_SIZE);
ArrayList<Tuple2<Long,ArrayList<MyDTO>>> extOrRepList = null;
extOrRepList = jdbcTemplate.query(sql,
extOrRepMapResultSetExtractor);
if (extOrRepList != null) {
iterator = extOrRepList.iterator();
}
return iterator;
}
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Missing-records-Ignite-cache-size-grows-tp10809p10966.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Missing records Ignite cache size grows
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi diopek,
I can't see how cache size relates to value classes and value object
content.
I rework my test and still can't reproduce this issue even with 2 millions
of entries in cache, but you mention about a half a million.
On Fri, Mar 3, 2017 at 6:25 PM, diopek <de...@gmail.com> wrote:
> Hi Andrew,
> Difference in your repro;
> your cache entry is <Long, ArrayList<String>>,
> and you always add single entry to value ArrayList
> in our case
> <Long, ArrayList<MyPOJO>>
> MyPOJO is java bean that has ~30 attributes.
> and ArrayList might have sometimes 100s of objects.
> During my local tests, that I am using ~20GB of RAM, I couldn't reproduce
> either.
> Issue occurs with high number of records with production data on servers
> with larger amount of RAM.
> I am wondering if you can try to replicate that scenario on your end.
> Thanks
>
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Missing-records-Ignite-cache-size-
> grows-tp10809p11022.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: Missing records Ignite cache size grows
Posted by diopek <de...@gmail.com>.
Hi Andrew,
Difference in your repro;
your cache entry is <Long, ArrayList<String>>,
and you always add single entry to value ArrayList
in our case
<Long, ArrayList<MyPOJO>>
MyPOJO is java bean that has ~30 attributes.
and ArrayList might have sometimes 100s of objects.
During my local tests, that I am using ~20GB of RAM, I couldn't reproduce
either.
Issue occurs with high number of records with production data on servers
with larger amount of RAM.
I am wondering if you can try to replicate that scenario on your end.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Missing-records-Ignite-cache-size-grows-tp10809p11022.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Missing records Ignite cache size grows
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
I've made a repro and it works fine for me.
Please check if I missed smth?
On Wed, Mar 1, 2017 at 5:15 PM, diopek <de...@gmail.com> wrote:
> Below is our implementation of LoadOnlyCacheStore inputIterator method.
> As we tested many times there is no issue on uniqueness of generated cache
> keys.
> But still after loading high number of records into IgniteCache, records
> count doesn't match, and few 10s of keys were missing out of million
> records. Also if we use the Java cache populated just before serializing
> this cache into IgniteCache, all number of records are matching.
>
> In short, after several days of try-outs and debugging, we narrowed down
> the
> root cause of issue to LoadOnlyCacheStore base class. So, at this point,
> we
> need Ignite core team support to resolve this bug.
> Thanks
>
>
> Yes, we checked uniqueness of key several times. Below is representative
> snippet of our code, how we generate cache key inside inputIterator method.
>
> @Override
> protected Iterator<Tuple2<Long,ArrayList<MyDTO>>>*
> inputIterator*(Object... args) throws CacheLoaderException {
> Iterator<Tuple2<Long,ArrayList<MyDTO>>> iterator = null;
>
> ResultSetExtractor<ArrayList<Tuple2<Long,ArrayList<MyDTO>>>>
> extOrRepMapResultSetExtractor = new
> ResultSetExtractor<ArrayList<Tuple2<Long,ArrayList<MyDTO>>>>() {
> @Override
> public ArrayList<Tuple2<Long,ArrayList<MyDTO>>>
> extractData(ResultSet rs)
> throws SQLException, DataAccessException {
> ArrayList<Tuple2<Long,ArrayList<MyDTO>>>
> extOrRepList = new
> ArrayList<Tuple2<Long,ArrayList<MyDTO>>>(startSize);
> *final AtomicLong entryCnt = new AtomicLong(0);*
> while (rs.next()) {
> extOrRepList.add(new Tuple2<Long,
> ArrayList<MyDTO>>(*entryCnt.incrementAndGet()*,prevDTOList));
> }
> return extOrRepList;
> }
> };
>
> jdbcTemplate.setFetchSize(SQL_FETCH_SIZE);
> ArrayList<Tuple2<Long,ArrayList<MyDTO>>> extOrRepList =
> null;
> extOrRepList = jdbcTemplate.query(sql,
> extOrRepMapResultSetExtractor);
>
> if (extOrRepList != null) {
> iterator = extOrRepList.iterator();
> }
> return iterator;
>
> }
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Missing-records-Ignite-cache-size-
> grows-tp10809p10966.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov