Problem initializing cc2.8xlarge instances

[David Rosenstrauch <da...@darose.net>]

We've been pretty successful in using whirr to spin up clusters of AWS 
cc1.4xlarge (cluster compute) instances.  However, I'm now trying to 
have it spin up clusters of cc2.8xlarge instances, and it's failing. 
Would love to find a way to work around this.


Background:

We've rolled our own custom AMI (built off of the stock Ubuntu Precise 
AMI's at http://cloud.ubuntu.com/ami/), which bundles in some of our own 
software and data.  We're then able to spin up clusters of cc1.4xlarge 
machines using this AMI.

When I was initially building this AMI, one of the issues I ran into is 
that Ubuntu names the ephemeral drives differently than whirr is 
expecting.  Whirr is looking for drives /dev/sdb, sdc, etc., while under 
Ubuntu/Xen they're actually named /dev/xvdb, xvdc, etc.  However, this 
was easy to work around:  I just set up symlinks from /dev/xvdb to sdb 
and so on.


However, for some reason this solution isn't working on the cc2.8xlarge 
instances.  I've created an AMI with four ephemeral disks, and have set 
up symlinks for each of them (sdb -> xvdb, sdc -> xvdc, sdd -> xvdd, sde 
-> xvde).  But whirr isn't seeing the latter 2 disks.  Instead, it's 
trying to initialize /dev/sdb and sdc twice, resulting in the Hadoop 
daemons being unable to start successfully on these machines.

The ultimate culprit seems to be this line in one of the node 
initialization scripts:

prepare_all_disks 
'/data0,/dev/sdb;/data1,/dev/sdc;/data2,/dev/sdb;/data3,/dev/sdc'

But I have to admit, I have no idea why it's trying to initialize the 
disks that way.  I very definitely have 4 distinct disks (and symlinks) 
on these data node machines:

# ls -l /dev/xvd*
brw-rw---- 1 root disk 202, 16 Oct  2 20:53 /dev/xvdb
brw-rw---- 1 root disk 202, 32 Oct  2 20:53 /dev/xvdc
brw-rw---- 1 root disk 202, 48 Oct  2 20:53 /dev/xvdd
brw-rw---- 1 root disk 202, 64 Oct  2 20:53 /dev/xvde

# ls -l /dev/sd*
brw-rw---- 1 root disk 8, 0 Oct  2 20:53 /dev/sda
brw-rw---- 1 root disk 8, 1 Oct  2 20:53 /dev/sda1
lrwxrwxrwx 1 root root    4 Oct  2 20:53 /dev/sdb -> xvdb
lrwxrwxrwx 1 root root    4 Oct  2 20:53 /dev/sdc -> xvdc
lrwxrwxrwx 1 root root    4 Oct  2 20:53 /dev/sdd -> xvdd
lrwxrwxrwx 1 root root    4 Oct  2 20:53 /dev/sde -> xvde


Anyone have any idea what's happening here and/or how to fix?

Thanks,

DR

Re: Problem initializing cc2.8xlarge instances

[Amlan <am...@yahoo.com>]

DR,

Did you find a solution to this issue? 

I am hitting the identical issue (with CentOS 5.4)?

Thanks,
/ Amlan