Re: Node permanently in maintenance mode

[Sergey Chugunov <se...@gmail.com>]

Hello Piotr,

Let me explain the situation you've faced: on startup node detected that
some cache data files may be corrupted and proceeding with a normal
startup routine would lead to JVM crash.
That's why node stopped normal startup and entered maintenance mode.

In that situation (potential data files corruption) Ignite cannot fix
anything automatically and simply waits for the user to intervene and
delete files in question (with an option of backing them up before).

So you've done everything right to address the situation. But message in
logs definitely needs to be improved, I've filed a ticket for that [1].

[1] https://issues.apache.org/jira/browse/IGNITE-15268

Thank you,
Sergey.



On Fri, Jul 30, 2021 at 2:41 PM Piotr Jagielski <pj...@touk.pl> wrote:

> Ok I managed to start the node by deleting maintenance_tasks.mntc file and
> corrupted cache from node storage directory
>
> On 2021/07/30 11:09:01, Piotr Jagielski <pj...@touk.pl> wrote:
> > Hi,
> > We recently switched our cluster to 2.10.
> > We had problems with one of the caches before the upgrade, and deleted
> some data.
> > After the upgrade Ignite has run for 1-2 day and today one of the nodes
> stated in "Maintenance mode" after the restart:
> >
> > 2021-07-30 12:59:41 WARN  Maintenance task found, stop restoring memory
> > 2021-07-30 12:59:41 INFO  Node requires maintenance, non-empty set of
> maintenance tasks is found: [corrupted-cache-data-files-task]
> >
> > The problem is that this task never ends and the node cannot join the
> rest of the cluster.
> > Is there any option we can move to normal mode or cancel this task?
> >
> > Regards,
> > Piotr
> >
> >
>