You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "kevin.zheng" <zh...@huawei.com> on 2016/04/19 12:58:18 UTC
ignite problem when loading large amount of data into cache
Hi there,
I am trying to load a large table content into cache (about 10g).
I have initiate 8 nodes in two servers (each has 4 nodes).
My server has 48g and 32g memory respectively.
During the loading process, I have the following exception message, and all
the nodes stopped:
[14:19:15,322][SEVERE][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi]
TcpDiscov erSpi's message worker
thread failed abnormally. Stopping the node in order to p
revent cluster wide instability.
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDequ
e.java:522)
at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.jav
a:684)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.b
ody(ServerImpl.java:5779)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body
(ServerImpl.java:2161)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[14:19:15] Topology snapshot [ver=35, servers=1, clients=0, CPUs=8,
heap=1.0GB]
[14:19:15] Ignite node stopped OK [uptime=00:36:09:525]
[14:19:19] Topology snapshot [ver=30, servers=6, clients=0, CPUs=16,
heap=6.0GB]
[14:23:08,016][SEVERE][tcp-disco-msg-worker-#2%null%][TcpDiscoverySpi]
TcpDiscov erSpi's message worker
thread failed abnormally. Stopping the node in order to p
revent cluster wide instability.
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDequ
e.java:522)
at
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.jav
a:684)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.b
ody(ServerImpl.java:5779)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body
(ServerImpl.java:2161)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
any idea how to fix it?
best regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: re: ignite problem when loading large amount of data into cache
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Kevin,
Normally restarted Ignite node will pull data from other nodes. We call
this process "rebalance".
Also Ignite comes with a special store type - local store (see
@CacheLocalStore annotation). Local store could be used to re-load node's
data after restart. However, Ignite doesn't have any local store
implementations. You can find one in some Ignite-based commercial products,
such as GridGain.
Vladimir.
On Wed, Apr 20, 2016 at 9:39 AM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi Val,
> When the exception occurred, I checked the forum and reset the java vm
> size to 30g, and also I split my table into 10 smaller tables, each
> contains 1g data.
> In this case, I saw your suggestion on off-heap settings. I don't want to
> reload all the data again, So I asked if it is possible to make the
> configuration take effect immediately at running time.
>
> Btw, I see there are backup settings in partition mode, like this
> <property name="backups" value="1"/>
> But I did not see where those backups are stored. Is there any settings
> like redis which will automatically load dump files when redis recovered
> from an accident?
>
> Regards,
> Kevin
>
> -----邮件原件-----
> 发件人: vkulichenko [mailto:valentin.kulichenko@gmail.com]
> 发送时间: 2016年4月20日 12:42
> 收件人: user@ignite.apache.org
> 主题: Re: re: ignite problem when loading large amount of data into cache
>
> Kevin,
>
> Configuration of the existing cache can't be changed in runtime. The only
> option is to destroy the cache and create it again with new parameters (you
> will lose all in-memory data, of course). What's the use case when you can
> need this?
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4347.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
re: re: ignite problem when loading large amount of data into cache
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Val,
When the exception occurred, I checked the forum and reset the java vm size to 30g, and also I split my table into 10 smaller tables, each contains 1g data.
In this case, I saw your suggestion on off-heap settings. I don't want to reload all the data again, So I asked if it is possible to make the configuration take effect immediately at running time.
Btw, I see there are backup settings in partition mode, like this <property name="backups" value="1"/>
But I did not see where those backups are stored. Is there any settings like redis which will automatically load dump files when redis recovered from an accident?
Regards,
Kevin
-----邮件原件-----
发件人: vkulichenko [mailto:valentin.kulichenko@gmail.com]
发送时间: 2016年4月20日 12:42
收件人: user@ignite.apache.org
主题: Re: re: ignite problem when loading large amount of data into cache
Kevin,
Configuration of the existing cache can't be changed in runtime. The only option is to destroy the cache and create it again with new parameters (you will lose all in-memory data, of course). What's the use case when you can need this?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: re: ignite problem when loading large amount of data into cache
Posted by vkulichenko <va...@gmail.com>.
Kevin,
Configuration of the existing cache can't be changed in runtime. The only
option is to destroy the cache and create it again with new parameters (you
will lose all in-memory data, of course). What's the use case when you can
need this?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignite problem when loading large amount of data into cache
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Val, should we document it in the FAQ section?
On Tue, Apr 19, 2016 at 5:59 PM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi Val,
> Thank you for your suggestion.
> I have one more question: is it possible to change the server
> configuration on running time? let's say I follow the off-heap-memory
> suggestions and set the off-heap size, how to make the server
> configuration takes effect immediately?
>
>
> Cheers,
> Kevin
>
> -----邮件原件-----
> 发件人: vkulichenko [mailto:valentin.kulichenko@gmail.com]
> 发送时间: 2016年4月20日 4:48
> 收件人: user@ignite.apache.org
> 主题: Re: ignite problem when loading large amount of data into cache
>
> Hi Kevin,
>
> With such large heap sizes it's very likely that you had long GC pauses
> that caused the topology to fall apart. To avoid this I would recommend to
> allocate not more that 10-12G of heap and store the data in off-heap caches
> [1].
>
> [1] https://apacheignite.readme.io/docs/off-heap-memory
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4340.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
re: ignite problem when loading large amount of data into cache
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Val,
Thank you for your suggestion.
I have one more question: is it possible to change the server configuration on running time? let's say I follow the off-heap-memory suggestions and set the off-heap size, how to make the server configuration takes effect immediately?
Cheers,
Kevin
-----邮件原件-----
发件人: vkulichenko [mailto:valentin.kulichenko@gmail.com]
发送时间: 2016年4月20日 4:48
收件人: user@ignite.apache.org
主题: Re: ignite problem when loading large amount of data into cache
Hi Kevin,
With such large heap sizes it's very likely that you had long GC pauses that caused the topology to fall apart. To avoid this I would recommend to allocate not more that 10-12G of heap and store the data in off-heap caches [1].
[1] https://apacheignite.readme.io/docs/off-heap-memory
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4340.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: ignite problem when loading large amount of data into cache
Posted by vkulichenko <va...@gmail.com>.
Hi Kevin,
With such large heap sizes it's very likely that you had long GC pauses that
caused the topology to fall apart. To avoid this I would recommend to
allocate not more that 10-12G of heap and store the data in off-heap caches
[1].
[1] https://apacheignite.readme.io/docs/off-heap-memory
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/ignite-problem-when-loading-large-amount-of-data-into-cache-tp4324p4340.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.