You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ray <ra...@cisco.com> on 2017/09/19 07:51:31 UTC
Ignite YARN deployment mode issues
I'm using ignite 2.1, my ignite config xml is as follows.
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true"/>
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-arg type="java.lang.String"
value="log4j2.xml"/>
</bean>
</property>
<property name="persistentStoreConfiguration">
<bean
class="org.apache.ignite.configuration.PersistentStoreConfiguration"/>
</property>
And in standalone mode, ignite can read the log4j2.xml in local directory
and works ok.
When deployed in YARN, ignite says it can find log4j2.xml with a bunch of
spring exceptions.
My cluster.properties is as follows
IGNITE_NODE_COUNT=6
IGNITE_RUN_CPU_PER_NODE=6
IGNITE_MEMORY_PER_NODE=30000
IGNITE_VERSION=2.1.0
IGNITE_PATH=/***/apache-ignite-fabric-2.1.0.zip
IGNITE_XML_CONFIG=/***/ignite-config/default-config.xml
IGNITE_USERS_LIBS=/***/ignite-libs/
I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not
working.
And another question is how to activate Ignite with persistent store enabled
in YARN mode.
I tried running ./control.sh --host ignitenode --port 11211 --activate
And the log says.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl <init>
WARNING: Failed to initialize topology on client start. Will retry in
background.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl <init>
INFO: Client started [id=5f00be2b-7679-46e7-9f8e-3435f7f1d759, protocol=TCP]
Something fail during activation, exception message: Latest topology update
failed.
Sep 19, 2017 6:25:27 AM
org.apache.ignite.internal.client.impl.GridClientImpl stop
INFO: Client stopped [id=5f00be2b-7679-46e7-9f8e-3435f7f1d759,
waitCompletion=true]
And there's no log in the server node side.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN deployment mode issues
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello Ray,
Yes, please try it and tell if it helps.
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN deployment mode issues
Posted by Ray <ra...@cisco.com>.
Hi Ilya
Do you mean modify the original config from
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-arg type="java.lang.String"
value="log4j2.xml"/>
</bean>
</property>
to
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-arg type="java.lang.String"
value="bin/log4j2.xml"/>
</bean>
</property>
if my understanding is correct?
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN deployment mode issues
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Ray,
Did you try specifying "bin/log4j2.xml" as constructor-arg then? In
stand-alone deployment, it could save you that Ignite also looks in current
directory in addition to IGNITE_HOME
--
Ilya Kasnacheev
2017-09-20 16:57 GMT+03:00 Ray <ra...@cisco.com>:
> Thanks for the reply, Ilya.
>
> The IGNITE_HOME for yarn is
> /yarn/nm/usercache/username/appcache/application_appid/
> container_containerID/ignite/
>
> The log4j2.xml is under IGNITE_HOME/bin folder too.
> If I launch ignite using ignist.sh, it's working fine and the log4j2.xml
> under IGNITE_HOME/bin folder can be picked up by ignite.
> But for YARN deployment, the issue still exists.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite YARN deployment mode issues
Posted by Ray <ra...@cisco.com>.
Thanks for the reply, Ilya.
The IGNITE_HOME for yarn is
/yarn/nm/usercache/username/appcache/application_appid/container_containerID/ignite/
The log4j2.xml is under IGNITE_HOME/bin folder too.
If I launch ignite using ignist.sh, it's working fine and the log4j2.xml
under IGNITE_HOME/bin folder can be picked up by ignite.
But for YARN deployment, the issue still exists.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN deployment mode issues
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello Ray!
--
Ilya Kasnacheev
2017-09-19 10:51 GMT+03:00 Ray <ra...@cisco.com>:
> I'm using ignite 2.1, my ignite config xml is as follows.
>
> I tried to put the log4j2.xml file under IGNITE_USERS_LIBS, still not
> working.
>
I guess it should be relative to IGNITE_HOME, or absolute path (I expect
it's no-no with YARN).
What is IGNITE_HOME exactly in case of YARN, I will try to find out.
--
Ilya Kasnacheev
Re: Ignite YARN deployment mode issues
Posted by Ray <ra...@cisco.com>.
Figured the second question out by myself.
The node I'm running ./control.sh does not seem to have ignite grid started.
Here's the yarn log.
[08:09:30,866][SEVERE][main][IgniteKernal] Exception during start
processors, node will be stopped and close connections
class org.apache.ignite.IgniteException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2931)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:374)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:696)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:929)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: java.io.FileNotFoundException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2925)
... 16 more
[08:09:30,868][SEVERE][main][IgniteKernal] Got exception while starting
(will rollback startup routine).
class org.apache.ignite.IgniteException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2931)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:374)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:696)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:929)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: java.io.FileNotFoundException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2925)
... 16 more
[08:09:30,873][SEVERE][main][IgniteKernal] Failed to pre-stop processor:
GridProcessorAdapter []
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.access$300(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.unLock(GridCacheDatabaseSharedManager.java:559)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.onKernalStop0(GridCacheDatabaseSharedManager.java:580)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:120)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:983)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2124)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2072)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1078)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
[08:09:30] Ignite node stopped wih ERRORS [uptime=00:00:00:808]
class org.apache.ignite.IgniteException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2931)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2899)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:374)
at
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:696)
at
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1788)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:929)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1896)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1648)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1076)
at
org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:994)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:880)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:779)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:618)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at
org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: java.io.FileNotFoundException:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.<init>(GridCacheDatabaseSharedManager.java:2925)
... 16 more
Failed to start grid:
/yarn/nm/usercache/root/appcache/application_1505700561210_0016/container_1505700561210_0016_01_000076/ignite/apache-ignite-fabric-2.1.0-bin/work/db/10_29_42_49_127_0_0_1_47500/lock
(Permission denied)
I ran the Ignite by the command of sudo hadoop jar ignite-yarn-2.1.0.jar
./ignite-yarn-2.1.0.jar cluster.properties
Why I get permission denied when I run this command as root?
One thing worse mention is that the node didn't start Ignite is the node I
submit the Ignite yarn application.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/