You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Pascal <pa...@gmail.com> on 2017/08/02 11:07:17 UTC
MemoryPolicy size vs cache size
Hi,
I try to use Ignite 2.1 to simulate the situation where a table doesn't fit
in memory and data is read from disk every time a full table scan occurs.
I've adapted the PersistentStoreExample to create a cache of a million items
which is stored in 200MB of data files and 120MB of index files. I call
PersistentStoreConfiguration.setFileIOFactory() to register a custom
FileIOFactory that tracks what file I/O is occurring. This tracker shows the
files are read during startup of the example, but not while the SQL query is
running. The cache shouldn't fit in 10MB and I would expect that during the
query the data would be read from disk. Why is this not happening and what
do I need to change to test this?
Thanks for your help.
Pascal
<bean class="org.apache.ignite.configuration.MemoryConfiguration">
<property name="pageSize" value="2048"/>
<property name="systemCacheInitialSize" value="#{10 * 1024 * 1024}"/>
<property name="systemCacheMaxSize" value="#{10 * 1024 * 1024}"/>
<property name="defaultMemoryPolicyName" value="default_mem_plc"/>
<property name="memoryPolicies">
<list>
<bean class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
<property name="name" value="default_mem_plc"/>
<property name="initialSize" value="#{10 * 1024 * 1024}"/>
<property name="maxSize" value="#{10 * 1024 * 1024}"/>
</bean>
</list>
</property>
</bean>
public class PersistentStoreExample {
private static final boolean UPDATE = false;
private static final long COUNT = 1000000;
public static void doMain() throws Exception {
Ignition.setClientMode(false);
Ignite ig =
Ignition.start("D:/Ignite/2.1/examples/config/persistentstore/example-persistent-store.xml");
IgniteConfiguration ic = ig.configuration();
PersistentStoreConfiguration pc =
ic.getPersistentStoreConfiguration();
pc.setPersistentStorePath("C:\\tmp\\");
MyFileIOFactory myFact = new MyFileIOFactory();
pc.setFileIOFactory(myFact);
ig.active(true);
IgniteCache<Long, MyPerson> cache = ig.cache("personCache");
if (UPDATE) {
System.out.println("Populating the cache...");
try (IgniteDataStreamer<Long, MyPerson> streamer =
ig.dataStreamer("personCache")) {
streamer.allowOverwrite(true);
for (long i = 0; i < COUNT; i++) {
MyPerson o = new MyPerson(i, "per-" + i);
streamer.addData(i, o);
if (i > 0 && i % 10_000 == 0) System.out.println("Done:
" + i);
}
}
}
while(true)
{
Thread.sleep(3000);
System.out.println("\nFull scan " + COUNT + " times...");
for(int getId = 0; getId< COUNT; getId++) {
QueryCursor cur = cache.query(new SqlFieldsQuery("select id,
name from MyPerson where name like ?").setArgs("per-" + getId));
if(getId % 100000 == 0) {
System.out.println(" SQL Result: " + cur.getAll());
}
}
}
}
}
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: MemoryPolicy size vs cache size
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Pascal,
How would you imagine to cache query results if the resultset doesn't fit
to memory ?
You can try to increase max memory size in memory policy cache used.
On Thu, Aug 3, 2017 at 6:25 PM, Pascal <pa...@gmail.com> wrote:
> Hi Andrey,
>
> thanks for your response. I actually do call getAll() 10 times every
> iteration. If I call getAll() 1.000.000 times like you suggest I do see the
> I/O I would expect. Is is possible the results for the 10 SQL queries are
> cached somehow?
>
> Anyway, this solves my problem.
> Thanks again.
> Pascal
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892p15951.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: MemoryPolicy size vs cache size
Posted by Pascal <pa...@gmail.com>.
Hi Andrey,
thanks for your response. I actually do call getAll() 10 times every
iteration. If I call getAll() 1.000.000 times like you suggest I do see the
I/O I would expect. Is is possible the results for the 10 SQL queries are
cached somehow?
Anyway, this solves my problem.
Thanks again.
Pascal
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892p15951.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: MemoryPolicy size vs cache size
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Pascal,
Looks like you do not actually run a query. Ignite doesn't fetch full
resultset by default and use paging for fetching.
Try to make *cursor.getAll() *called on every iteration
or (which is better) get iterator and iterate over rows to reduce heap
memory pressure.
On Wed, Aug 2, 2017 at 2:07 PM, Pascal <pa...@gmail.com> wrote:
> Hi,
>
> I try to use Ignite 2.1 to simulate the situation where a table doesn't fit
> in memory and data is read from disk every time a full table scan occurs.
> I've adapted the PersistentStoreExample to create a cache of a million
> items
> which is stored in 200MB of data files and 120MB of index files. I call
> PersistentStoreConfiguration.setFileIOFactory() to register a custom
> FileIOFactory that tracks what file I/O is occurring. This tracker shows
> the
> files are read during startup of the example, but not while the SQL query
> is
> running. The cache shouldn't fit in 10MB and I would expect that during the
> query the data would be read from disk. Why is this not happening and what
> do I need to change to test this?
>
> Thanks for your help.
> Pascal
>
> <bean class="org.apache.ignite.configuration.MemoryConfiguration">
> <property name="pageSize" value="2048"/>
> <property name="systemCacheInitialSize" value="#{10 * 1024 *
> 1024}"/>
> <property name="systemCacheMaxSize" value="#{10 * 1024 * 1024}"/>
> <property name="defaultMemoryPolicyName" value="default_mem_plc"/>
> <property name="memoryPolicies">
> <list>
> <bean class="org.apache.ignite.configuration.
> MemoryPolicyConfiguration">
> <property name="name"
> value="default_mem_plc"/>
> <property name="initialSize" value="#{10 *
> 1024 * 1024}"/>
> <property name="maxSize" value="#{10 *
> 1024 * 1024}"/>
> </bean>
> </list>
> </property>
> </bean>
>
> public class PersistentStoreExample {
> private static final boolean UPDATE = false;
> private static final long COUNT = 1000000;
> public static void doMain() throws Exception {
> Ignition.setClientMode(false);
> Ignite ig =
> Ignition.start("D:/Ignite/2.1/examples/config/persistentstore/example-
> persistent-store.xml");
> IgniteConfiguration ic = ig.configuration();
> PersistentStoreConfiguration pc =
> ic.getPersistentStoreConfiguration();
> pc.setPersistentStorePath("C:\\tmp\\");
> MyFileIOFactory myFact = new MyFileIOFactory();
> pc.setFileIOFactory(myFact);
> ig.active(true);
> IgniteCache<Long, MyPerson> cache = ig.cache("personCache");
>
> if (UPDATE) {
> System.out.println("Populating the cache...");
> try (IgniteDataStreamer<Long, MyPerson> streamer =
> ig.dataStreamer("personCache")) {
> streamer.allowOverwrite(true);
> for (long i = 0; i < COUNT; i++) {
> MyPerson o = new MyPerson(i, "per-" + i);
> streamer.addData(i, o);
> if (i > 0 && i % 10_000 == 0) System.out.println("Done:
> " + i);
> }
> }
> }
>
> while(true)
> {
> Thread.sleep(3000);
> System.out.println("\nFull scan " + COUNT + " times...");
> for(int getId = 0; getId< COUNT; getId++) {
> QueryCursor cur = cache.query(new SqlFieldsQuery("select
> id,
> name from MyPerson where name like ?").setArgs("per-" + getId));
> if(getId % 100000 == 0) {
> System.out.println(" SQL Result: " + cur.getAll());
> }
> }
> }
> }
> }
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/MemoryPolicy-size-vs-cache-size-tp15892.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov