[jira] Updated: (WHIRR-55) Users should be able to override an arbitrary Hadoop property before launch

["Tom White (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/WHIRR-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated WHIRR-55:
---------------------------

    Attachment: WHIRR-55.patch

Here's an initial attempt at this (for the Java implementation). Configuration is generated by a HadoopConfigurationBuilder, and is pushed to a file on cluster nodes using jclouds' Statements.createFile call.

HadoopConfigurationBuilder takes care of dynamic properties like fs.default.name and mapred.job.tracker which depend on the cluster object. It may be extended in future to set mapred.reduce.tasks according to the number of slots in the cluster, or mapred.tasktracker.{map,reduce}.tasks.maximum according to the number of CPUs on each instance.

Properties may be overridden by specifying them in the Whirr configuration. For example, to override Hadoop's dfs.replication property to 2 you would add

{code}
hadoop-hdfs.dfs.replication=2
{code}

to your Whirr properties file. The hadoop-hdfs prefix signifies that the property should go in hdfs-site.xml. (This patch also incorporates WHIRR-149.)

As a simplification, this patch also removes the webserver running on the namenode, since the URLs for the namenode and jobtracker are now logged explicitly:

{noformat}
Namenode web UI available at http://ec2-184-73-89-144.compute-1.amazonaws.com:50070
Jobtracker web UI available at http://ec2-184-73-89-144.compute-1.amazonaws.com:50030
{noformat}

so you can go directly to the web UIs.

> Users should be able to override an arbitrary Hadoop property before launch
> ---------------------------------------------------------------------------
>
>                 Key: WHIRR-55
>                 URL: https://issues.apache.org/jira/browse/WHIRR-55
>             Project: Whirr
>          Issue Type: Improvement
>          Components: service/hadoop
>            Reporter: Tom White
>             Fix For: 0.3.0
>
>         Attachments: WHIRR-55.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.