[jira] [Commented] (WHIRR-330) BYON doesn't work with HadoopConfigurationBuilder

["Tom White (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049608#comment-13049608 ] 

Tom White commented on WHIRR-330:
---------------------------------

We could also make HadoopConfigurationBuilder more defensive so that it falls back to Hadoop defaults if Hardware is null. This is probably the simplest fix.

Fix 1 seems like a good thing to do too, but it requires a jclouds change.

> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
>                 Key: WHIRR-330
>                 URL: https://issues.apache.org/jira/browse/WHIRR-330
>             Project: Whirr
>          Issue Type: Bug
>          Components: core, service/hadoop
>    Affects Versions: 0.5.0
>         Environment: CentOS 5.5
>            Reporter: John Amos
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
>         at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
>         at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
>         at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
>         at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
>         at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
>         at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
>         at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
>         at org.apache.whirr.cli.Main.run(Main.java:68)
>         at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
>         .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
>   coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
>   String publicIp = Iterables.get(node.getPublicAddresses(), 0);
>   return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute.  It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira