You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Oscar Torreno <os...@shapelets.io> on 2019/07/08 07:15:44 UTC
IGFS block at startup
Hi,
I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.
Have you experienced such issue before? If yes, is there any workaround to overcome it?
Best regards, thanks in advance.
Oscar
[Logo]<https://www.shapelets.io/>
Oscar Torreno
Software Engineer
m: + 34 675 026 952
e: oscar.torreno@shapelets.io<ma...@shapelets.io>
C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
[LinkedIn icon]<https://www.linkedin.com/company/shapelets/> [Twitter icon] <https://twitter.com/shapelets>
Re: IGFS block at startup
Posted by Oscar Torreno <os...@shapelets.io>.
Hello Ilya,
Please find attached the log and the complete stack traces of both nodes.
I n this case the node “shapelets-2” was the one blocked at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.awaitInit(IgfsMetaManager.java:177)
Best,
--
Oscar Torreno
From: Ilya Kasnacheev <il...@gmail.com>
Reply-To: "user@ignite.apache.org" <us...@ignite.apache.org>
Date: Monday, 8 July 2019 at 14:15
To: "user@ignite.apache.org" <us...@ignite.apache.org>
Subject: Re: IGFS block at startup
Hello!
It is hard to say what is happening here without full stack trace from all threads of both nodes. Can you provide that?
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 13:48, Oscar Torreno <os...@shapelets.io>>:
Hello Ilya,
Please find attached the docker compose log of both nodes (shapelets-1 and shapelets-2). Shapelets-2 was the one able to start without problems in this case. Attaching the output of jstack for the main Thread of the shapelets-1 node.
Regards,
--
Oscar Torreno
From: Ilya Kasnacheev <il...@gmail.com>>
Reply-To: "user@ignite.apache.org<ma...@ignite.apache.org>" <us...@ignite.apache.org>>
Date: Monday, 8 July 2019 at 11:25
To: "user@ignite.apache.org<ma...@ignite.apache.org>" <us...@ignite.apache.org>>
Subject: Re: IGFS block at startup
Hello!
Can you please provide complete logs and stack traces from both nodes?
I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <os...@shapelets.io>>:
Hi,
I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.
Have you experienced such issue before? If yes, is there any workaround to overcome it?
Best regards, thanks in advance.
Oscar
[Logo]<https://www.shapelets.io/>
Oscar Torreno
Software Engineer
m: + 34 675 026 952
e: oscar.torreno@shapelets.io<ma...@shapelets.io>
C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
[LinkedIn icon]<https://www.linkedin.com/company/shapelets/> [Twitter icon] <https://twitter.com/shapelets>
Re: IGFS block at startup
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It is hard to say what is happening here without full stack trace from all
threads of both nodes. Can you provide that?
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 13:48, Oscar Torreno <os...@shapelets.io>:
> Hello Ilya,
>
>
>
> Please find attached the docker compose log of both nodes (shapelets-1 and
> shapelets-2). Shapelets-2 was the one able to start without problems in
> this case. Attaching the output of jstack for the main Thread of the
> shapelets-1 node.
>
>
>
> Regards,
>
> *--*
>
> *Oscar Torreno*
>
>
>
> *From: *Ilya Kasnacheev <il...@gmail.com>
> *Reply-To: *"user@ignite.apache.org" <us...@ignite.apache.org>
> *Date: *Monday, 8 July 2019 at 11:25
> *To: *"user@ignite.apache.org" <us...@ignite.apache.org>
> *Subject: *Re: IGFS block at startup
>
>
>
> Hello!
>
>
>
> Can you please provide complete logs and stack traces from both nodes?
>
>
>
> I guess we have a lot of tests where we start several IGFS nodes and they
> finish just fine.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <os...@shapelets.io>:
>
> Hi,
>
>
>
> I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using
> docker-compose) with 2 IGFS configured. When I start both nodes at the same
> time, almost always one of them starts without problems, but the second one
> hangs at line 120 of the IgfsMetaManager class (doing an await on a
> CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of
> race condition/inconsistent state problem because of the simultaneous start.
>
>
>
> Have you experienced such issue before? If yes, is there any workaround to
> overcome it?
>
>
>
> Best regards, thanks in advance.
>
> Oscar
>
>
>
> [image: Logo] <https://www.shapelets.io/>
>
> *Oscar Torreno*
>
> Software Engineer
>
> m: + 34 675 026 952
>
> e: oscar.torreno@shapelets.io
>
> C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
>
> [image: LinkedIn icon] <https://www.linkedin.com/company/shapelets/> [image:
> Twitter icon] <https://twitter.com/shapelets>
>
>
>
>
Re: IGFS block at startup
Posted by Oscar Torreno <os...@shapelets.io>.
Hello Ilya,
Please find attached the docker compose log of both nodes (shapelets-1 and shapelets-2). Shapelets-2 was the one able to start without problems in this case. Attaching the output of jstack for the main Thread of the shapelets-1 node.
Regards,
--
Oscar Torreno
From: Ilya Kasnacheev <il...@gmail.com>
Reply-To: "user@ignite.apache.org" <us...@ignite.apache.org>
Date: Monday, 8 July 2019 at 11:25
To: "user@ignite.apache.org" <us...@ignite.apache.org>
Subject: Re: IGFS block at startup
Hello!
Can you please provide complete logs and stack traces from both nodes?
I guess we have a lot of tests where we start several IGFS nodes and they finish just fine.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <os...@shapelets.io>>:
Hi,
I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using docker-compose) with 2 IGFS configured. When I start both nodes at the same time, almost always one of them starts without problems, but the second one hangs at line 120 of the IgfsMetaManager class (doing an await on a CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of race condition/inconsistent state problem because of the simultaneous start.
Have you experienced such issue before? If yes, is there any workaround to overcome it?
Best regards, thanks in advance.
Oscar
[Logo]<https://www.shapelets.io/>
Oscar Torreno
Software Engineer
m: + 34 675 026 952
e: oscar.torreno@shapelets.io<ma...@shapelets.io>
C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
[LinkedIn icon]<https://www.linkedin.com/company/shapelets/> [Twitter icon] <https://twitter.com/shapelets>
Re: IGFS block at startup
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you please provide complete logs and stack traces from both nodes?
I guess we have a lot of tests where we start several IGFS nodes and they
finish just fine.
Regards,
--
Ilya Kasnacheev
пн, 8 июл. 2019 г. в 10:16, Oscar Torreno <os...@shapelets.io>:
> Hi,
>
>
>
> I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using
> docker-compose) with 2 IGFS configured. When I start both nodes at the same
> time, almost always one of them starts without problems, but the second one
> hangs at line 120 of the IgfsMetaManager class (doing an await on a
> CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of
> race condition/inconsistent state problem because of the simultaneous start.
>
>
>
> Have you experienced such issue before? If yes, is there any workaround to
> overcome it?
>
>
>
> Best regards, thanks in advance.
>
> Oscar
>
>
>
> [image: Logo] <https://www.shapelets.io/>
>
> *Oscar Torreno*
>
> Software Engineer
>
> m: + 34 675 026 952
>
> e: oscar.torreno@shapelets.io
>
> C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
>
> [image: LinkedIn icon] <https://www.linkedin.com/company/shapelets/> [image:
> Twitter icon] <https://twitter.com/shapelets>
>
>
>
Re: IGFS block at startup
Posted by Denis Magda <dm...@apache.org>.
Oscar,
IGFS won't longer be supported by the community:
http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42405.html
Switch to the solutions of this kind if you're accelerating Hadoop:
https://docs.gridgain.com/docs/bdb-getting-started
-
Denis
On Mon, Jul 8, 2019 at 12:16 AM Oscar Torreno <os...@shapelets.io>
wrote:
> Hi,
>
>
>
> I am trying to start a fresh 2 nodes Ignite 2.7.0 cluster (using
> docker-compose) with 2 IGFS configured. When I start both nodes at the same
> time, almost always one of them starts without problems, but the second one
> hangs at line 120 of the IgfsMetaManager class (doing an await on a
> CountDownLatch). Rarely, both nodes progress, so it seems to be a kind of
> race condition/inconsistent state problem because of the simultaneous start.
>
>
>
> Have you experienced such issue before? If yes, is there any workaround to
> overcome it?
>
>
>
> Best regards, thanks in advance.
>
> Oscar
>
>
>
> [image: Logo] <https://www.shapelets.io/>
>
> *Oscar Torreno*
>
> Software Engineer
>
> m: + 34 675 026 952
>
> e: oscar.torreno@shapelets.io
>
> C/ Puerta del Mar 18, 2º. 29005, Málaga,Spain
>
> [image: LinkedIn icon] <https://www.linkedin.com/company/shapelets/> [image:
> Twitter icon] <https://twitter.com/shapelets>
>
>
>