[jira] [Comment Edited] (AMBARI-3074) Ambari wont start NodeManager because one of multiple folders not created

["Sumit Mohanty (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AMBARI-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757997#comment-13757997 ] 

Sumit Mohanty edited comment on AMBARI-3074 at 9/4/13 5:28 PM:
---------------------------------------------------------------

LGTM, +1. We should think about the scenario where none of the folders get created. In that case NM will start but will fail after start. Consider adding validation and error reporting as a feature.
                
      was (Author: sumitmohanty):
    LGTM. We should think about the scenario where none of the folders get created. In that case NM will start but will fail after start. Consider adding validation and error reporting as a feature.
                  
> Ambari wont start NodeManager because one of multiple folders not created
> -------------------------------------------------------------------------
>
>                 Key: AMBARI-3074
>                 URL: https://issues.apache.org/jira/browse/AMBARI-3074
>             Project: Ambari
>          Issue Type: Bug
>          Components: agent
>    Affects Versions: 1.4.0
>            Reporter: Siddharth Wagle
>            Assignee: Siddharth Wagle
>             Fix For: 1.4.1
>
>         Attachments: AMBARI-3074.patch
>
>
> {{yarn-site}} having:
> {noformat}
> "yarn.nodemanager.local-dirs" : "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
> "yarn.nodemanager.log-dirs" : "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn",
> {noformat}
> Now {{/grid/3}} was mounted as read-only due to some disk errors. Though other folders got successfully created, Ambari will not start the NodeManager process.
> {noformat}
> notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: mkdir: cannot create directory `/grid/3/hadoop': Read-only file system
> err: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: change from notrun to 0 failed: mkdir -p /grid/3/hadoop/yarn returned 1 instead of one of [0] at /var/lib/ambari-agent/puppet/modules/hdp/manifests/init.pp:479
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/content: content changed '{md5}e5d17c21c7a5e1db9f3af35cba71df0a' to '{md5}2ca1d267a46f1aecac726caabaa16774'
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/owner: owner changed 'hdfs' to 'yarn'
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[core-site]/File[/etc/hadoop/conf/core-site.xml]/content: content changed '{md5}86d742a780d59a957ea0a283dec03784' to '{md5}8506e4402ba8140ea4f9fed97b6f94e2'
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[yarn-site]/File[/etc/hadoop/conf/yarn-site.xml]/content: content changed '{md5}d84a967ce47a6b77734ed8f53d817c6e' to '{md5}42940cca6e8f64ae5de50524fb131274'
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::begin]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::begin]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::begin]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c  'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true
> warning: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]: Skipping because of failed dependencies
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}093cb1899b3c3b9dc4a7c1c93729c18b' to '{md5}4c462999cc47e6f6ba0e6381d71d81ba'
> notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/owner: owner changed 'mapred' to 'yarn'
> notice: Finished catalog run in 2.39 seconds
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira