sql query large data set which is larger than memory

[November <yu...@126.com>]

Hi

I have a large data set larger than memory.  I want to do some sql operation
on it. So I enabled swap space. But the program still run out of memory.
Below is my cache configuration code. Is there any thing wrong?

Thanks

var cache = ignite.GetOrCreateCache<object, object>(new CacheConfiguration
                {
                    Name = CacheName,
                    EnableSwap = true
                });



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-large-data-set-which-is-larger-than-memory-tp6219.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: sql query large data set which is larger than memory

[vkulichenko <va...@gmail.com>]

Hi,

Swap space means that whatever data you need to execute the query needs to
be unswapped first. In other words, if you do query like 'select * from', it
will not help. But if it has a reasonable condition that will allow result
set to fit in memory along with all other data, you will not run out of
memory.

But generally using swap space is a bad practice. If you don't have enough
RAM, it's better to add it, otherwise you lose much of Ignite advantages.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-large-data-set-which-is-larger-than-memory-tp6219p6242.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: sql query large data set which is larger than memory

[November <yu...@126.com>]

off-heap memory is also in RAM. The data set is larger than RAM. 
If it's possible to put cache in swap space and do sql operations while some
cache data store in swap space?



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-large-data-set-which-is-larger-than-memory-tp6219p6227.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: sql query large data set which is larger than memory

[AndreyVel <an...@gmail.com>]

HI November,

Please look at memory parameters JVM_OPTS for server nodes.
Try to use off-heap memory for cache [1] 

[1] http://apacheignite.gridgain.org/v1.5-b1/docs/off-heap-memory



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-large-data-set-which-is-larger-than-memory-tp6219p6224.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.