You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Thomas Kramer <do...@gmx.de> on 2021/03/23 17:36:52 UTC
Correctly stopping/deactivating server node
Hi,
when running Ignite inside a docker container, and the container is
being stopped, what is the safest way to handle shutdown event?
Currently I'm using a ShutdownHook to the Java runtime and try to
quickly deactivate and disconnect the cluster. But I get an exception
like this:
class org.apache.ignite.IgniteException: Failed to execute deactivate
request, node is stopping.
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
at
org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
at
mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
at mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
at
mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
execute deactivate request, node is stopping.
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
at
org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
... 4 more
If I only do a disconnect without deactivate then it works well. But my
understanding is the last server node should also deactivate the
cluster? How can this be achieved in such Docker environment?
Thanks,
Thomas.
Re: Re: Correctly stopping/deactivating server node
Posted by do...@gmx.de.
Yes it works well indeed! Thanks!
On 05.04.21 at 12:56, Ilya Kasnacheev wrote:
From: "Ilya Kasnacheev" <il...@gmail.com>
Date: 5. April 2021
To: user@ignite.apache.org
Cc:
Subject: Re: Correctly stopping/deactivating server node
Hello!
Yes it should be safe, provided that your own hook works correctly.
Regards,
\--
Ilya Kasnacheev
вс, 4 апр. 2021 г. в 23:18, DonTequila
<[don.tequila@gmx.de](mailto:don.tequila@gmx.de)>:
> Actually I found that this is indeed true. Ignite supports the JRE option:
>
> -DIGNITE_NO_SHUTDOWN_HOOK=true
>
> When I set this my app shutdown hook has enough time to call deactive the
> cluster.
>
> I wonder if this is safe to do?
>
>
>
> \--
> Sent from: <http://apache-ignite-users.70518.x6.nabble.com/>
>
Re: Correctly stopping/deactivating server node
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Yes it should be safe, provided that your own hook works correctly.
Regards,
--
Ilya Kasnacheev
вс, 4 апр. 2021 г. в 23:18, DonTequila <do...@gmx.de>:
> Actually I found that this is indeed true. Ignite supports the JRE option:
>
> -DIGNITE_NO_SHUTDOWN_HOOK=true
>
> When I set this my app shutdown hook has enough time to call deactive the
> cluster.
>
> I wonder if this is safe to do?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Correctly stopping/deactivating server node
Posted by DonTequila <do...@gmx.de>.
Actually I found that this is indeed true. Ignite supports the JRE option:
-DIGNITE_NO_SHUTDOWN_HOOK=true
When I set this my app shutdown hook has enough time to call deactive the
cluster.
I wonder if this is safe to do?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Correctly stopping/deactivating server node
Posted by DonTequila <do...@gmx.de>.
No, this is happening for the last server node with no other nodes connected.
To me it looks like Ignite itself is already handling its own shutdown hook
when SIGTERM is sent and then internally starts disconnect. So Ignite
doesn't allow deactivate any more. Is this possible true?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Correctly stopping/deactivating server node
Posted by Shubham Agrawal <ag...@gmail.com>.
I guess, basically you are trying to see, which java process is stopping at
last. If this is the last process, you have to deactivate the cluster, or
else just disconnect. You might be getting the above error as one node is
trying to disconnect and another node is trying to deactivate.
In order to resolve this, you can try out a centralised approach to check
how many processes are running. For instance, zookeeper. If a new node is
spawning, we ideally add a Z node in the zookeeper and remove it if it gets
deleted. If you don't want to use a zookeeper, you can have another
approach like files for different processes in a centralised location. You
can add a file on spawning and delete when getting killed. In the shutdown
hook, every process checks how many files are there in the centralised
location or how many nodes are there in the zookeeper. If the value is 1,
deactivate the cluster, else only disconnect. There could be other
approaches, but this is something you could try.
On Sat, Apr 3, 2021 at 12:01 PM DonTequila <do...@gmx.de> wrote:
> My standalone Java application is executed as process 1 in the Docker
> container. At start it activates the cluster which has persistence enabled.
> I'm not using the Ignite script bin/ignite.sh.
>
> "docker stop" sends a SIGTERM to my java application. My application
> shutdown hook gets called where I try to deactivate and then disconnect the
> Ignite instance. This works well when only disconnecting, but doesn't work
> when deactivating the cluster first. Exception says "Failed to execute
> deactivate request, node is stopping." Full stacktrace in my original post.
>
> Docker waits 10 seconds for a graceful stop of the process before it sends
> a
> SIGKILL.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Correctly stopping/deactivating server node
Posted by DonTequila <do...@gmx.de>.
My standalone Java application is executed as process 1 in the Docker
container. At start it activates the cluster which has persistence enabled.
I'm not using the Ignite script bin/ignite.sh.
"docker stop" sends a SIGTERM to my java application. My application
shutdown hook gets called where I try to deactivate and then disconnect the
Ignite instance. This works well when only disconnecting, but doesn't work
when deactivating the cluster first. Exception says "Failed to execute
deactivate request, node is stopping." Full stacktrace in my original post.
Docker waits 10 seconds for a graceful stop of the process before it sends a
SIGKILL.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Correctly stopping/deactivating server node
Posted by Shubham Agrawal <ag...@gmail.com>.
We need to see how the docker container is shutting down. Like what is the
order of the services to shut down.
If you are shutting down the ignite service before shutting down the Java
process, we might encounter the above error you mentioned.
Vice Versa, this problem should not come. You can try this and check out if
this works.
Regards,
Shubham Agrawal
On Thu, Apr 1, 2021 at 10:58 AM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> I guess that it's already too late to deactivate if the node is shutting
> down.
>
> What are scenarios for which you are trying to prepare here?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 31 мар. 2021 г. в 22:13, DonTequila <do...@gmx.de>:
>
>> Has anyone been able to look at this and give me some advise? Thanks!
>>
>>
>> DonTequila wrote
>> > Hi,
>> >
>> > when running Ignite inside a docker container, and the container is
>> > being stopped, what is the safest way to handle shutdown event?
>> >
>> > Currently I'm using a ShutdownHook to the Java runtime and try to
>> > quickly deactivate and disconnect the cluster. But I get an exception
>> > like this:
>> >
>> > class org.apache.ignite.IgniteException: Failed to execute deactivate
>> > request, node is stopping.
>> > at
>> >
>> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
>> > at
>> >
>> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
>> > at
>> >
>> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
>> > at
>> > mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
>> > at
>> > mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
>> > at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
>> > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
>> > execute deactivate request, node is stopping.
>> > at
>> >
>> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
>> > at
>> >
>> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
>> > at
>> >
>> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
>> > at
>> >
>> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
>> > ... 4 more
>> >
>> > If I only do a disconnect without deactivate then it works well. But my
>> > understanding is the last server node should also deactivate the
>> > cluster? How can this be achieved in such Docker environment?
>> >
>> > Thanks,
>> > Thomas.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: Correctly stopping/deactivating server node
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I guess that it's already too late to deactivate if the node is shutting
down.
What are scenarios for which you are trying to prepare here?
Regards,
--
Ilya Kasnacheev
ср, 31 мар. 2021 г. в 22:13, DonTequila <do...@gmx.de>:
> Has anyone been able to look at this and give me some advise? Thanks!
>
>
> DonTequila wrote
> > Hi,
> >
> > when running Ignite inside a docker container, and the container is
> > being stopped, what is the safest way to handle shutdown event?
> >
> > Currently I'm using a ShutdownHook to the Java runtime and try to
> > quickly deactivate and disconnect the cluster. But I get an exception
> > like this:
> >
> > class org.apache.ignite.IgniteException: Failed to execute deactivate
> > request, node is stopping.
> > at
> >
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
> > at
> >
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
> > at
> >
> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
> > at
> > mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
> > at
> > mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
> > at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
> > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> > execute deactivate request, node is stopping.
> > at
> >
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
> > at
> >
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
> > at
> >
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
> > at
> >
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
> > ... 4 more
> >
> > If I only do a disconnect without deactivate then it works well. But my
> > understanding is the last server node should also deactivate the
> > cluster? How can this be achieved in such Docker environment?
> >
> > Thanks,
> > Thomas.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Correctly stopping/deactivating server node
Posted by DonTequila <do...@gmx.de>.
Has anyone been able to look at this and give me some advise? Thanks!
DonTequila wrote
> Hi,
>
> when running Ignite inside a docker container, and the container is
> being stopped, what is the safest way to handle shutdown event?
>
> Currently I'm using a ShutdownHook to the Java runtime and try to
> quickly deactivate and disconnect the cluster. But I get an exception
> like this:
>
> class org.apache.ignite.IgniteException: Failed to execute deactivate
> request, node is stopping.
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1067)
> at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:317)
> at
> mediafan.net.ignite.IgniteHelper.setClusterActiveStatus(IgniteHelper.java:238)
> at
> mediafan.client.helper.ConsoleApp.setActive(ConsoleApp.java:370)
> at
> mediafan.client.helper.ConsoleApp.disconnect(ConsoleApp.java:342)
> at mediafan.client.helper.ConsoleApp$1.run(ConsoleApp.java:70)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
> execute deactivate request, node is stopping.
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState0(GridClusterStateProcessor.java:1007)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:832)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.changeGlobalState(GridClusterStateProcessor.java:820)
> at
> org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:314)
> ... 4 more
>
> If I only do a disconnect without deactivate then it works well. But my
> understanding is the last server node should also deactivate the
> cluster? How can this be achieved in such Docker environment?
>
> Thanks,
> Thomas.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/