[jira] [Created] (MYRIAD-128) Issue with Flex down, Pending NMs stuck in staging and don't get to active task.

["Sarjeet Singh (JIRA)" <ji...@apache.org>]

Sarjeet Singh created MYRIAD-128:
------------------------------------

             Summary: Issue with Flex down, Pending NMs stuck in staging and don't get to active task.
                 Key: MYRIAD-128
                 URL: https://issues.apache.org/jira/browse/MYRIAD-128
             Project: Myriad
          Issue Type: Bug
          Components: Scheduler
    Affects Versions: Myriad 0.1.0
            Reporter: Sarjeet Singh


Seeing some issue when I tried flexing NMs from Myriad UI. On flexing down active NM,  pending NMs doesn't go to active state (not sowing in 'Active Tasks') and there is no active NM showing on Myriad UI. Although, there is a NM running on the node (verified from jps). 

mapr     20528 20526  1 17:23 ?        00:00:26 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/bin/java -Dproc_nodemanager -Xmx1000m -Dhadoop.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs -Dyarn.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs -Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dhadoop.root.logger=INFO,console -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native -Dyarn.policy.file=hadoop-policy.xml -server -Dnodemanager.resource.io-spindles=4.0 -Dyarn.resourcemanager.hostname=testrm.marathon.mesos -Dyarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor -Dnodemanager.resource.cpu-vcores=0 -Dnodemanager.resource.memory-mb=0 -Dmyriad.yarn.nodemanager.address=0.0.0.0:31000 -Dmyriad.yarn.nodemanager.localizer.address=0.0.0.0:31001 -Dmyriad.yarn.nodemanager.webapp.address=0.0.0.0:31002 -Dmyriad.mapreduce.shuffle.port=0.0.0.0:31003 -Dhadoop.login=maprsasl -Dhttps.protocols=TLSv1.2 -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider -Dhadoop.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs -Dyarn.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs -Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.log -Dyarn.home.dir=/opt/mapr/hadoop/hadoop-2.7.0 -Dhadoop.home.dir=/opt/mapr/hadoop/hadoop-2.7.0 -Dhadoop.root.logger=INFO,console -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/mapr/hadoop/hadoop-2.7.0/lib/native -classpath /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/nm-config/log4j.properties:/opt/mapr/lib/JPam-1.1.jar org.apache.hadoop.yarn.server.nodemanager.NodeManager

>From myriad UI:

Active Tasks
Killable Tasks
Pending Tasks
Staging Tasks
nm.large.123badb1-57d8-4bd2-aa2e-de9fc1898c7f
nm.medium.f2c4126c-4cb2-46af-a1e0-690034b914b8
nm.medium.a9e9fd84-350a-48bc-bcd2-8712ecdc8c66
nm.medium.663f9c6e-f28e-4395-8540-70c306eb04c5
nm.medium.93f7cc91-9263-48a7-821e-3b0ffbe70e66

This is the state even after waited for about 30 min or so after flexing down the NM.

I tried this on a single node cluster though, but looks like the problem can happen in any case.

I started RM from marathon and was able to get RM & Myriad up & running. With RM launched, there is a CGS (medium profile) NM is launched along with it as well which is shown as 'Active Task' on Myriad UI. Then, I launched some large profile & zero profile NM which are shown now in 'Pending tasks' since there is a (CGS default) NM already running on a single node cluster.

Then, I tried flexing down NM from myriad UI, which flexed up the active NM and all pending NMs start moving to staging tasks, and then they stuck in staging task for longer time. On waited for about > 30min, I dont see any active task for NM and all of the pending NM tasks are shown in 'Staging task' only. (See the screenshot)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)