You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "John Amos (JIRA)" <ji...@apache.org> on 2011/06/15 05:04:47 UTC
[jira] [Created] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
BYON doesn't work with HadoopConfigurationBuilder
-------------------------------------------------
Key: WHIRR-330
URL: https://issues.apache.org/jira/browse/WHIRR-330
Project: Whirr
Issue Type: Bug
Components: service/hadoop
Affects Versions: 0.5.0
Environment: CentOS 5.5
Reporter: John Amos
I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
{noformat}
bin/whirr launch-cluster --config recipes/hadoop-byon.properties
{noformat}
I see the following error:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:68)
at org.apache.whirr.cli.Main.main(Main.java:106)
{noformat}
The code looks like this:
{code:java}
Hardware hardware = Iterables.getFirst(taskTrackers, null)
.getNodeMetadata().getHardware();
int coresPerNode = 0;
for (Processor processor : hardware.getProcessors()) {
coresPerNode += processor.getCores();
}
{code}
The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
{code:java}
public Instance apply(NodeMetadata node) {
String publicIp = Iterables.get(node.getPublicAddresses(), 0);
return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
}
{code}
I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
(1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
(2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "John Amos (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Amos updated WHIRR-330:
----------------------------
Component/s: core
Description:
I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
{noformat}
bin/whirr launch-cluster --config recipes/hadoop-byon.properties
{noformat}
I see the following error:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:68)
at org.apache.whirr.cli.Main.main(Main.java:106)
{noformat}
The code looks like this:
{code:java}
Hardware hardware = Iterables.getFirst(taskTrackers, null)
.getNodeMetadata().getHardware();
int coresPerNode = 0;
for (Processor processor : hardware.getProcessors()) {
coresPerNode += processor.getCores();
}
{code}
The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
{code:java}
public Instance apply(NodeMetadata node) {
String publicIp = Iterables.get(node.getPublicAddresses(), 0);
return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
}
{code}
I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
(1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
(2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
was:
I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
{noformat}
bin/whirr launch-cluster --config recipes/hadoop-byon.properties
{noformat}
I see the following error:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:68)
at org.apache.whirr.cli.Main.main(Main.java:106)
{noformat}
The code looks like this:
{code:java}
Hardware hardware = Iterables.getFirst(taskTrackers, null)
.getNodeMetadata().getHardware();
int coresPerNode = 0;
for (Processor processor : hardware.getProcessors()) {
coresPerNode += processor.getCores();
}
{code}
The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
{code:java}
public Instance apply(NodeMetadata node) {
String publicIp = Iterables.get(node.getPublicAddresses(), 0);
return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
}
{code}
I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
(1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
(2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-330:
------------------------------
Attachment: WHIRR-330.patch
Attached trivial patch as discussed. I agree that we should fix this in jclouds sooner or later.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
> Fix For: 0.6.0
>
> Attachments: WHIRR-330.patch
>
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049608#comment-13049608 ]
Tom White commented on WHIRR-330:
---------------------------------
We could also make HadoopConfigurationBuilder more defensive so that it falls back to Hadoop defaults if Hardware is null. This is probably the simplest fix.
Fix 1 seems like a good thing to do too, but it requires a jclouds change.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-330:
------------------------------
Fix Version/s: 0.6.0
Assignee: Andrei Savu
Status: Patch Available (was: Open)
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
> Assignee: Andrei Savu
> Fix For: 0.6.0
>
> Attachments: WHIRR-330.patch
>
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050571#comment-13050571 ]
Tom White commented on WHIRR-330:
---------------------------------
+1 looks good.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
> Assignee: Andrei Savu
> Fix For: 0.6.0
>
> Attachments: WHIRR-330.patch
>
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-330) BYON doesn't work with
HadoopConfigurationBuilder
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-330:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this. Thanks John for the extended bug description.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
> Assignee: Andrei Savu
> Fix For: 0.6.0
>
> Attachments: WHIRR-330.patch
>
>
> I created a hadoop-byon.properties file in recipes similar to recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(), null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109 because the NodeMetadata has a null Hardware attribute. It seems like one of two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of hardware, including processors and cores per processor (and the first NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira