Configuration problems

[Christian Stucchio <st...@gmail.com>]

Greetings,

I'm trying to get started using Whirr to set up a hadoop cluster on ec2. I'm
using version 0.3.0-cdh3u0 (0.3.0, installed via Cloudera's ubuntu
repositories), so I apologize in advance if I should be asking Cloudera
rather than this list.

I've set up a cluster using this configuration (taken from here:
http://www.cloudera.com/blog/2011/01/map-reduce-with-ruby-using-apache-hadoop/
,
but adding the dfs.permissions=false configuration option):

whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn+dn,1 dn+tt
whirr.provider=ec2
whirr.location-id=ap-southeast-1
whirr.identity=<EC2 USER ID>
whirr.credential=<EC2 SECRET>
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
hadoop-hdfs.dfs.permissions=false
hadoop-hdfs.dfs.replication=2

Then I run these commands:
$ export HADOOP_CONF_DIR=~/.whirr/myhadoopcluster/
$ hadoop fs -mkdir /test
11/04/28 08:01:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied:
user=stucchio, access=WRITE, inode="":hdfs:supergroup:rwxr-xr-x

This is just the standard hadoop "permission denied" error, suggesting that
I did something wrong in my configuration. Logging in to the cluster shows
me that no configuration options were actually written:

$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

</configuration>

$ cat slaves
localhost

So it looks as if I don't even have a hadoop cluster, just two independent
boxes in the cloud with hadoop installed.

Any idea what I'm doing wrong? Is there a better set of docs I should be
reading from?



Thanks,


Chris

Re: Configuration problems

[Andrei Savu <sa...@gmail.com>]

You should be using the recently released Whirr 0.4.0. Download it from:

http://www.apache.org/dyn/closer.cgi/incubator/whirr/

and check the "recipes" folder. The properties files found there are
well documented.

In Whirr 0.3.0 it's not that easy to change Hadoop configuration
parameters (WHIRR-55 [1] was fixed in 0.4.0).

[1] https://issues.apache.org/jira/browse/WHIRR-55

Cheers,

-- Andrei Savu / andreisavu.ro

On Thu, Apr 28, 2011 at 4:22 PM, Christian Stucchio <st...@gmail.com> wrote:
> Greetings,
> I'm trying to get started using Whirr to set up a hadoop cluster on ec2. I'm
> using version 0.3.0-cdh3u0 (0.3.0, installed via Cloudera's ubuntu
> repositories), so I apologize in advance if I should be asking Cloudera
> rather than this list.
> I've set up a cluster using this configuration (taken from
> here: http://www.cloudera.com/blog/2011/01/map-reduce-with-ruby-using-apache-hadoop/ ,
> but adding the dfs.permissions=false configuration option):
> whirr.service-name=hadoop
> whirr.cluster-name=myhadoopcluster
> whirr.instance-templates=1 jt+nn+dn,1 dn+tt
> whirr.provider=ec2
> whirr.location-id=ap-southeast-1
> whirr.identity=<EC2 USER ID>
> whirr.credential=<EC2 SECRET>
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
> whirr.hadoop-install-runurl=cloudera/cdh/install
> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
> hadoop-hdfs.dfs.permissions=false
> hadoop-hdfs.dfs.replication=2
> Then I run these commands:
> $ export HADOOP_CONF_DIR=~/.whirr/myhadoopcluster/
> $ hadoop fs -mkdir /test
> 11/04/28 08:01:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
> in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> mkdir: org.apache.hadoop.security.AccessControlException: Permission denied:
> user=stucchio, access=WRITE, inode="":hdfs:supergroup:rwxr-xr-x
> This is just the standard hadoop "permission denied" error, suggesting that
> I did something wrong in my configuration. Logging in to the cluster shows
> me that no configuration options were actually written:
> $ cat hdfs-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <!-- Put site-specific property overrides in this file. -->
> <configuration>
> </configuration>
> $ cat slaves
> localhost
> So it looks as if I don't even have a hadoop cluster, just two independent
> boxes in the cloud with hadoop installed.
> Any idea what I'm doing wrong? Is there a better set of docs I should be
> reading from?
>
>
> Thanks,
>
> Chris
>