You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mamaco <ma...@163.com> on 2018/02/27 03:39:21 UTC
SortedEvictionPolicy doesn't work as expected
Here I have a five entries as below:
(14,4);
(21,1);
(32,2);
(113,3);
(15,5);
and I want to use SortedEvictionPolicy to keep below 3 entries (sort by
values in descending order and get top 3 items):
15-->5
14-->4
113-->3
The actual output is:
21-->1
32-->2
113-->3
14-->4
15-->5
issue 1: The order is wrong, the class CustomizedComparator doesn't seem to
work.
issue 2: there are only 3 entries are expected, but it returns 5 ones,
MaxSize doesn't work.
Did I miss anything?
The source code:
package IgniteTesting.Expiry;
import javax.cache.Cache.Entry;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.sorted.SortedEvictionPolicy;
import org.apache.ignite.cache.query.QueryCursor;
import org.apache.ignite.cache.query.SqlQuery;
import org.apache.ignite.configuration.CacheConfiguration;
public class Application
{
public static void main( String[] args )
{
try(Ignite ignite = Ignition.start("......\\test.xml")){
CacheConfiguration<Integer, Integer> cfg = new CacheConfiguration<>();
cfg.setName("foo");
cfg.setOnheapCacheEnabled(true);
cfg.setCacheMode(CacheMode.LOCAL);
SortedEvictionPolicy<Integer,Integer> SEP = new
SortedEvictionPolicy<>(3,new CustomizedComparator());
cfg.setEvictionPolicy(SEP);
cfg.setIndexedTypes(Integer.class, Integer.class);
try(IgniteCache<Integer, Integer> cache=ignite.getOrCreateCache(cfg)){
cache.put(14,4);
cache.put(21,1);
cache.put(32,2);
cache.put(113,3);
cache.put(15,5);
SqlQuery<Integer,Integer> sql = new SqlQuery<>(Integer.class, "select
* from Integer");
try (QueryCursor<Entry<Integer, Integer>> cursor =
cache.query(sql)) {
for (Entry<Integer, Integer> e : cursor)
System.out.println(e.getKey()+"-->"+e.getValue().toString());
}
}
}
}
}
package IgniteTesting.Expiry;
import java.io.Serializable;
import java.util.Comparator;
import org.apache.ignite.cache.eviction.EvictableEntry;
public class CustomizedComparator implements
Serializable,Comparator<EvictableEntry<Integer,Integer>> {
private static final long serialVersionUID = 4755938152431669886L;
public int compare(EvictableEntry<Integer,Integer> a,
EvictableEntry<Integer,Integer> b) {
return a.getValue() > b.getValue() ? -1 : a.getValue() ==
b.getValue() ? 0 : 1;
}
}
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by mamaco <ma...@163.com>.
Everything is clear now, our product version is ignite 1.7, now considering
2.3 migration, the whole conversation is very important to us, especially
the key point of how Off-Heap works.
Thank you so much.
Marco
Denis Mekhanikov wrote
> No, It's not like that.
>
> You can access any entry from any node, but access time will be different.
> If an entry is stored on a different node, Ignite will make a network
> request to that node and return it to you.
> If the entry is available locally, then the access time will be lower.
> And if you configure an oh-heap caching, and an entry is available
> locally,
> then the access time will be even lower.
>
> So, on-heap cache doesn't affect availability of records, it's just an
> internal enhancement to make access to some entries faster.
> If you enable it, you won't need to change any code, you will just get
> lower access time for some records, and higher memory consumption.
>
> And eviction policy is not determining, which entries are stored in the
> cache, but the entries, which have lower access time from local nodes.
>
> Denis
>
> чт, 1 мар. 2018 г. в 1:36, mamaco <
> mamaco@
> >:
>
>> So, that means even if I have a local cache with SortedEvictionPolicy
>> configured and OnHeap enabled,
>> it's still impossible to get the entries from a distributed partition
>> cache, because the source cache is not centralized.
>> And finally, the solution is to set a standalone SortedMap via
>> RemoteEventListener.
>>
>> Correct?
>>
>> Marco
>>
>> Denis Mekhanikov wrote
>> Marco, If you access some records, that are stored in the off-heap
>> memory,
>> then you have to wait, while Ignite is deserializing the data and copying
>> it to Java heap. But if the needed entry is already available in Java
>> heap,
>> Ignite doesn't have to perform these steps to return the result, it can
>> just give you what it already has. So, the idea behind a Java heap cache
>> is
>> to make access to some specific entries faster. The entries, that are
>> kept
>> on the heap will be accessed way faster, than the once, that are stored
>> in
>> off-heap memory. Note, that all of this is true only for the local
>> records.
>> If you have multiple nodes, and you do a lot of cross-node reads, then
>> Java
>> heap cache won't help you much. Denis ср, 28 февр. 2018 г. в 20:41,
>> mamaco <[hidden
>> email]
>> <http:///user/SendEmail.jtp?type=email&email=mamaco%40>>: > Hi
>> Denis, > > Thank you for the response, I appreciate it. > Yes, I agree
>> with
>> you, it could be done by various solutions, 'REST', > 'Event Listener' or
>> any standalone instance. > According to the new design you mentioned,
>> ignite stop the support of ' >
>> *setMemoryMode(CacheMemoryMode.ONHEAP_TIERED)*' use Off-Heap in default >
>> and force a switch setOnheapCacheEnabled in 2.0.0+ version, however, if
>> we
>> > can't get the entries straight from on-heap, what do on-heap policies
>> stand > for (sort/fifo/random). Because no matter what we do, it returns
>> the whole > thing. or the goal is just to make a faster cache? I don't
>> mean
>> to be > negative, I'm just curious about the truth under the hood. > >
>> Marco > ------------------------------ > Sent from the Apache Ignite
>> Users
>> mailing list archive >
>> <http://apache-ignite-users.70518.x6.nabble.com/>
>> at Nabble.com. >
>>
>>
>> ------------------------------
>> Sent from the Apache Ignite Users mailing list archive
>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by Denis Mekhanikov <dm...@gmail.com>.
No, It's not like that.
You can access any entry from any node, but access time will be different.
If an entry is stored on a different node, Ignite will make a network
request to that node and return it to you.
If the entry is available locally, then the access time will be lower.
And if you configure an oh-heap caching, and an entry is available locally,
then the access time will be even lower.
So, on-heap cache doesn't affect availability of records, it's just an
internal enhancement to make access to some entries faster.
If you enable it, you won't need to change any code, you will just get
lower access time for some records, and higher memory consumption.
And eviction policy is not determining, which entries are stored in the
cache, but the entries, which have lower access time from local nodes.
Denis
чт, 1 мар. 2018 г. в 1:36, mamaco <ma...@163.com>:
> So, that means even if I have a local cache with SortedEvictionPolicy
> configured and OnHeap enabled,
> it's still impossible to get the entries from a distributed partition
> cache, because the source cache is not centralized.
> And finally, the solution is to set a standalone SortedMap via
> RemoteEventListener.
>
> Correct?
>
> Marco
>
> Denis Mekhanikov wrote
> Marco, If you access some records, that are stored in the off-heap memory,
> then you have to wait, while Ignite is deserializing the data and copying
> it to Java heap. But if the needed entry is already available in Java heap,
> Ignite doesn't have to perform these steps to return the result, it can
> just give you what it already has. So, the idea behind a Java heap cache is
> to make access to some specific entries faster. The entries, that are kept
> on the heap will be accessed way faster, than the once, that are stored in
> off-heap memory. Note, that all of this is true only for the local records.
> If you have multiple nodes, and you do a lot of cross-node reads, then Java
> heap cache won't help you much. Denis ср, 28 февр. 2018 г. в 20:41, mamaco <[hidden
> email] <http:///user/SendEmail.jtp?type=email&email=mamaco%40>>: > Hi
> Denis, > > Thank you for the response, I appreciate it. > Yes, I agree with
> you, it could be done by various solutions, 'REST', > 'Event Listener' or
> any standalone instance. > According to the new design you mentioned,
> ignite stop the support of ' >
> *setMemoryMode(CacheMemoryMode.ONHEAP_TIERED)*' use Off-Heap in default >
> and force a switch setOnheapCacheEnabled in 2.0.0+ version, however, if we
> > can't get the entries straight from on-heap, what do on-heap policies
> stand > for (sort/fifo/random). Because no matter what we do, it returns
> the whole > thing. or the goal is just to make a faster cache? I don't mean
> to be > negative, I'm just curious about the truth under the hood. > >
> Marco > ------------------------------ > Sent from the Apache Ignite Users
> mailing list archive > <http://apache-ignite-users.70518.x6.nabble.com/>
> at Nabble.com. >
>
>
> ------------------------------
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
Re: SortedEvictionPolicy doesn't work as expected
Posted by mamaco <ma...@163.com>.
So, that means even if I have a local cache with SortedEvictionPolicy
configured and OnHeap enabled,
it's still impossible to get the entries from a distributed partition cache,
because the source cache is not centralized.
And finally, the solution is to set a standalone SortedMap via
RemoteEventListener.
Correct?
Marco
Denis Mekhanikov wrote
> Marco,If you access some records, that are stored in the off-heap memory,
> thenyou have to wait, while Ignite is deserializing the data and copying
> it toJava heap.But if the needed entry is already available in Java heap,
> Ignite doesn'thave to perform these steps to return the result, it can
> just give you whatit already has.So, the idea behind a Java heap cache is
> to make access to some specificentries faster. The entries, that are kept
> on the heap will be accessed wayfaster, than the once, that are stored in
> off-heap memory.Note, that all of this is true only for the local records.
> If you havemultiple nodes, and you do a lot of cross-node reads, then Java
> heap cachewon't help you much.Denisср, 28 февр. 2018 г. в 20:41, mamaco
> <
> mamaco@
> >:> Hi Denis,>> Thank you for the response, I appreciate it.> Yes, I
> agree with you, it could be done by various solutions, 'REST',> 'Event
> Listener' or any standalone instance.> According to the new design you
> mentioned, ignite stop the support of '>
> *setMemoryMode(CacheMemoryMode.ONHEAP_TIERED)*' use Off-Heap in default>
> and force a switch setOnheapCacheEnabled in 2.0.0+ version, however, if
> we> can't get the entries straight from on-heap, what do on-heap policies
> stand> for (sort/fifo/random). Because no matter what we do, it returns
> the whole> thing. or the goal is just to make a faster cache? I don't mean
> to be> negative, I'm just curious about the truth under the hood.>> Marco>
> ------------------------------> Sent from the Apache Ignite Users mailing
> list archive> <http://apache-ignite-users.70518.x6.nabble.com/> at
> Nabble.com.>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by Denis Mekhanikov <dm...@gmail.com>.
Marco,
If you access some records, that are stored in the off-heap memory, then
you have to wait, while Ignite is deserializing the data and copying it to
Java heap.
But if the needed entry is already available in Java heap, Ignite doesn't
have to perform these steps to return the result, it can just give you what
it already has.
So, the idea behind a Java heap cache is to make access to some specific
entries faster. The entries, that are kept on the heap will be accessed way
faster, than the once, that are stored in off-heap memory.
Note, that all of this is true only for the local records. If you have
multiple nodes, and you do a lot of cross-node reads, then Java heap cache
won't help you much.
Denis
ср, 28 февр. 2018 г. в 20:41, mamaco <ma...@163.com>:
> Hi Denis,
>
> Thank you for the response, I appreciate it.
> Yes, I agree with you, it could be done by various solutions, 'REST',
> 'Event Listener' or any standalone instance.
> According to the new design you mentioned, ignite stop the support of '
> *setMemoryMode(CacheMemoryMode.ONHEAP_TIERED)*' use Off-Heap in default
> and force a switch setOnheapCacheEnabled in 2.0.0+ version, however, if we
> can't get the entries straight from on-heap, what do on-heap policies stand
> for (sort/fifo/random). Because no matter what we do, it returns the whole
> thing. or the goal is just to make a faster cache? I don't mean to be
> negative, I'm just curious about the truth under the hood.
>
> Marco
> ------------------------------
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
Re: SortedEvictionPolicy doesn't work as expected
Posted by mamaco <ma...@163.com>.
Hi Denis,
Thank you for the response, I appreciate it.
Yes, I agree with you, it could be done by various solutions, 'REST', 'Event
Listener' or any standalone instance.
According to the new design you mentioned, ignite stop the support of
'/setMemoryMode(CacheMemoryMode.ONHEAP_TIERED)/' use Off-Heap in default and
force a switch setOnheapCacheEnabled in 2.0.0+ version, however, if we
can't get the entries straight from on-heap, what do on-heap policies stand
for (sort/fifo/random). Because no matter what we do, it returns the whole
thing. or the goal is just to make a faster cache? I don't mean to be
negative, I'm just curious about the truth under the hood.
Marco
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by Denis Mekhanikov <dm...@gmail.com>.
Marco,
There is no way to perform "get" operations on heap entries solely, as far
as I know.
Looks like you are trying to miss-use a Java heap cache.
What you need can be achieved by using some tree data structure like Java
TreeSet.
Every time you insert something into cache, you can update this data
structure, so it will contain top 100 entries.
Such updates can be performed inside a continuous query listener:
https://apacheignite.readme.io/docs/continuous-queries
So, you'll have all data stored in Ignite cache and top 100 entries stored
in your own data structure.
Denis
ср, 28 февр. 2018 г. в 2:04, mamaco <ma...@163.com>:
> After reading valentin's example
> <https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/eviction/CustomEvictionPolicyExample.java>
> and a user discussion
> <http://apache-ignite-users.70518.x6.nabble.com/Enforcing-data-to-be-stored-in-Heap-only-td15141.html>,
>
>
> I found what you mentioned is correct, in a later test, after rolling back
> the ignite from 2.3.0 to 1.7.0, then everything works normally.
>
> But, *'how to read on-heap entries directly via ignite 2.0+'* is still a
> major pain to me,
> because what I need is 'top 100 entries', is there any approach to do
> this?
>
> Marco
>
>
> ------------------------------
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
Re: SortedEvictionPolicy doesn't work as expected
Posted by mamaco <ma...@163.com>.
After reading valentin's example
<https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/eviction/CustomEvictionPolicyExample.java>
and a user discussion
<http://apache-ignite-users.70518.x6.nabble.com/Enforcing-data-to-be-stored-in-Heap-only-td15141.html>
,
I found what you mentioned is correct, in a later test, after rolling back
the ignite from 2.3.0 to 1.7.0, then everything works normally.
But, *'how to read on-heap entries directly via ignite 2.0+'* is still a
major pain to me,
because what I need is 'top 100 entries', is there any approach to do this?
Marco
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by mamaco <ma...@163.com>.
Hi Denis,
Thank you for the explanation, so the cause is about Ignite use both of
on-heap and off-heap to store the entries and unfortunately I read the
everything from off-heap, so question #1 is: *how could I read the on-heap
entries directly?*
*My use case* is to get Top 100 entries by sorting a particular column in
descending order, see below detail:
Assume I have a continuous stream coming in from kafka and saving to a
partition cache which is located in 4 nodes, I could use map-reduce like
function to realize 'order by *** limit 100' query, however, the cache
itself is really huge, there's no performance guaranteed for realtime user
query. so, A simple and on-going solution is expected.
Question #2: *I wonder if SortedEvictionPolicy could be the best practise,
will this work?*
Thanks again.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: SortedEvictionPolicy doesn't work as expected
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi!
Eviction policy specifies, which entries should be kept in on-heap memory
in deserialized format.
So, the access time will be the lowest for such entries, when read from the
local node.
But such entries are stored twice: once in off-heap and once in on-heap.
And evicted entries are stored in off-heap memory anyway, so you are able
to access such values, even when they are evicted from on-heap space.
Entries from off-heap memory can be evicted per page, based on last access
time. It can be configured using the following method:
*DataRegionConfiguration.setPageEvictionMode(...)*
You can find all this information and more in the documentation:
https://apacheignite.readme.io/docs/evictions
P.S.
Instances of Integer shouldn't be checked for equality, using == operator.
They should be either unboxed first, or checked using equals()
*CustomizedComparator.compare(...) *can be rewritten as follows:
return Integer.compare(b.getValue(), a.getValue());
Denis
вт, 27 февр. 2018 г. в 6:39, mamaco <ma...@163.com>:
> Here I have a five entries as below:
> (14,4);
> (21,1);
> (32,2);
> (113,3);
> (15,5);
> and I want to use SortedEvictionPolicy to keep below 3 entries (sort by
> values in descending order and get top 3 items):
> 15-->5
> 14-->4
> 113-->3
>
> The actual output is:
> 21-->1
> 32-->2
> 113-->3
> 14-->4
> 15-->5
>
> issue 1: The order is wrong, the class CustomizedComparator doesn't seem to
> work.
> issue 2: there are only 3 entries are expected, but it returns 5 ones,
> MaxSize doesn't work.
> Did I miss anything?
>
> The source code:
>
> package IgniteTesting.Expiry;
> import javax.cache.Cache.Entry;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.eviction.sorted.SortedEvictionPolicy;
> import org.apache.ignite.cache.query.QueryCursor;
> import org.apache.ignite.cache.query.SqlQuery;
> import org.apache.ignite.configuration.CacheConfiguration;
>
> public class Application
> {
> public static void main( String[] args )
> {
> try(Ignite ignite = Ignition.start("......\\test.xml")){
> CacheConfiguration<Integer, Integer> cfg = new
> CacheConfiguration<>();
> cfg.setName("foo");
> cfg.setOnheapCacheEnabled(true);
> cfg.setCacheMode(CacheMode.LOCAL);
> SortedEvictionPolicy<Integer,Integer> SEP = new
> SortedEvictionPolicy<>(3,new CustomizedComparator());
> cfg.setEvictionPolicy(SEP);
> cfg.setIndexedTypes(Integer.class, Integer.class);
>
> try(IgniteCache<Integer, Integer>
> cache=ignite.getOrCreateCache(cfg)){
> cache.put(14,4);
> cache.put(21,1);
> cache.put(32,2);
> cache.put(113,3);
> cache.put(15,5);
>
> SqlQuery<Integer,Integer> sql = new
> SqlQuery<>(Integer.class, "select
> * from Integer");
> try (QueryCursor<Entry<Integer, Integer>>
> cursor =
> cache.query(sql)) {
> for (Entry<Integer, Integer> e : cursor)
>
> System.out.println(e.getKey()+"-->"+e.getValue().toString());
> }
>
>
> }
> }
>
> }
> }
>
>
> package IgniteTesting.Expiry;
>
> import java.io.Serializable;
> import java.util.Comparator;
> import org.apache.ignite.cache.eviction.EvictableEntry;
>
> public class CustomizedComparator implements
> Serializable,Comparator<EvictableEntry<Integer,Integer>> {
> private static final long serialVersionUID = 4755938152431669886L;
>
> public int compare(EvictableEntry<Integer,Integer> a,
> EvictableEntry<Integer,Integer> b) {
> return a.getValue() > b.getValue() ? -1 : a.getValue() ==
> b.getValue() ? 0 : 1;
> }
> }
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>