[jira] [Created] (BIGTOP-2453) nodemanager role requires common_mapred_app

["Kevin W Monroe (JIRA)" <ji...@apache.org>]

Kevin W Monroe created BIGTOP-2453:
--------------------------------------

             Summary: nodemanager role requires common_mapred_app
                 Key: BIGTOP-2453
                 URL: https://issues.apache.org/jira/browse/BIGTOP-2453
             Project: Bigtop
          Issue Type: Bug
          Components: deployment
    Affects Versions: 1.1.0
            Reporter: Kevin W Monroe
            Priority: Minor


When deploying a standalone nodemanager, puppet apply fails in a couple different ways:

First, /etc/hadoop/conf/core-site.xml is not created by the nodemanager class (or any of its prereqs). When trying to install a standalone nodemanger with the following in site.yaml:

'bigtop::roles_enabled': true
'bigtop::roles': nodemanager

The puppet apply will fail like this:
...
Error: Could not find dependency File[/etc/hadoop/conf/core-site.xml] for Service[hadoop-yarn-nodemanager] at /home/ubuntu/bigtop.release/bigtop-1.1.0/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:841
...

At first glance, it may seem like we just need to add common_hdfs (the creator of core-site.xml) to the nodemanager class.  However, I tried that and ran up against a second problem (longer trace at http://paste.ubuntu.com/16637532/):
...
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.mapred.ShuffleHandler not found
...

The problem here is that yarn-site.xml is specifying a missing mapred shuffle class, which prevents nodemanager from starting.  The solution for me was to include common_mapred_app in the nodemanager class (patch coming shortly).

This solves both problems with one line since common_mapred_app installs the missing mapred shuffle class and includes common_hdfs to create core-site.xml.

This may not be a common problem since I suspect most people are combining multiple slave roles (datanode, mapred-app, nodemanager) onto a single machine, but it would bite anyone that wanted to isolate the nodemanager role by itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)