You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Justin Ji <bi...@foxmail.com> on 2019/01/28 08:56:05 UTC
The cluster with persistence enable is stuck for 60 seconds when
restarting a node. Is this normal?
I have a cluster(2.7.0) with persistence enabled, the cluster has three
server nodes(8c16g) and every node has 8 Gb data(1 primary and 1 backup).
When I restart one of them, the whole cluster will stop about 60 seconds and
do not process any request from the client.
From the logs, I saw that the node is processing partition map exchange,
almost 60 seconds was spent on it.
My disk is SSD which has a high read and write speed.
The network is well too, the nodes can connect with each other, and the
latency is very low(less than 0.2 ms) when restarting.
I also have investigated the GC logs of three nodes, most of GC is finished
in 50ms.
My cache AtomicityMode is CacheAtomicityMode.ATOMIC, so there are no long
transactions.
So I want to ask is it normal that the cluster is stuck for 60 seconds?
If it is not normal, then where should the problem be most, how to optimize?
Looking forward to your replies.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds
when restarting a node. Is this normal?
Posted by Justin Ji <bi...@foxmail.com>.
Sergey -
Thank for your reply.
Yes, I defined consistentId for every node. Is it relates to partition map
exchange?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds when
restarting a node. Is this normal?
Posted by Sergey Antonov <an...@gmail.com>.
Hi!
"After I reduced the checkpointFrequency, the block time decrease to 10
seconds."
Good news!
"But I do not understand how the checkpointFrequency impacts the partition
map exchange, can someone explain it?"
Applying chages from WAL since last checkpoint (i.e. recovery) is a part of
PME proccess. So, the more often checkpoint the less changes will be
applied from WAL. You could find more information about checkopointing here
[1].
[1] https://apacheignite.readme.io/docs/persistence-checkpointing
ср, 30 янв. 2019 г. в 11:48, Justin Ji <bi...@foxmail.com>:
> Thank Sergey!
>
> From the email that he sent to me(I do not know why the email doesn't
> display in this post), I know that the default checkpointFrequency(180000)
> was too long, so it impacts the partition map exchange. After I reduced the
> checkpointFrequency, the block time decrease to 10 seconds.
>
> But I do not understand how the checkpointFrequency impacts the partition
> map exchange, can someone explain it?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
BR, Sergey Antonov
Re: The cluster with persistence enable is stuck for 60 seconds
when restarting a node. Is this normal?
Posted by Justin Ji <bi...@foxmail.com>.
Thank Sergey!
From the email that he sent to me(I do not know why the email doesn't
display in this post), I know that the default checkpointFrequency(180000)
was too long, so it impacts the partition map exchange. After I reduced the
checkpointFrequency, the block time decrease to 10 seconds.
But I do not understand how the checkpointFrequency impacts the partition
map exchange, can someone explain it?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds
when restarting a node. Is this normal?
Posted by Justin Ji <bi...@foxmail.com>.
Sergey -
Thank for your reply.
Yes, I can find them in the logs, I attached the logs of three nodes, the
node that was restarted is node3.
BTW, I use the default Rebalance Mode(ASYNC), so the rebalancing may not
impact the whole cluster.
Am I getting it wrong?
ignite-003.log
<http://apache-ignite-users.70518.x6.nabble.com/file/t2000/ignite-003.log>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds when
restarting a node. Is this normal?
Posted by Sergey Antonov <an...@gmail.com>.
Justin, may be problem in rebalancing?
https://apacheignite.readme.io/docs/rebalancing
Could you grep log messages by GridCachePartitionsExchangeManager with text
"Skipping rebalancing" or "Rebalancing scheduled" ?
пн, 28 янв. 2019 г. в 15:04, Justin Ji <bi...@foxmail.com>:
> Can someone give me some suggestions?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
BR, Sergey Antonov
Re: The cluster with persistence enable is stuck for 60 seconds
when restarting a node. Is this normal?
Posted by Justin Ji <bi...@foxmail.com>.
Can someone give me some suggestions?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds
when restarting a node. Is this normal?
Posted by Justin Ji <bi...@foxmail.com>.
Sergey -
Thank for your reply.
Yes, I defined consistentId for every node. Does it relates to partition map
exchange?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The cluster with persistence enable is stuck for 60 seconds when
restarting a node. Is this normal?
Posted by Sergey Antonov <an...@gmail.com>.
Hi!
Did you define consistentId for nodes?
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/IgniteConfiguration.html#setConsistentId-java.io.Serializable-
пн, 28 янв. 2019 г. в 11:56, Justin Ji <bi...@foxmail.com>:
> I have a cluster(2.7.0) with persistence enabled, the cluster has three
> server nodes(8c16g) and every node has 8 Gb data(1 primary and 1 backup).
>
> When I restart one of them, the whole cluster will stop about 60 seconds
> and
> do not process any request from the client.
>
> From the logs, I saw that the node is processing partition map exchange,
> almost 60 seconds was spent on it.
>
> My disk is SSD which has a high read and write speed.
> The network is well too, the nodes can connect with each other, and the
> latency is very low(less than 0.2 ms) when restarting.
> I also have investigated the GC logs of three nodes, most of GC is finished
> in 50ms.
> My cache AtomicityMode is CacheAtomicityMode.ATOMIC, so there are no long
> transactions.
>
> So I want to ask is it normal that the cluster is stuck for 60 seconds?
>
> If it is not normal, then where should the problem be most, how to
> optimize?
>
> Looking forward to your replies.
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
BR, Sergey Antonov