You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2010/09/21 18:49:32 UTC

[jira] Created: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

EBS cluster doesn't work with CDH3 on Lucid
-------------------------------------------

                 Key: WHIRR-96
                 URL: https://issues.apache.org/jira/browse/WHIRR-96
             Project: Whirr
          Issue Type: Bug
          Components: contrib/python
            Reporter: Tom White


Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ] 

Zach Bailey commented on WHIRR-96:
----------------------------------

Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{{

Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None
}}

However, the next part where the scripts wait for the jobtracker to start time out:

{{
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

{{

ep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist
}}

> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:57 PM:
------------------------------------------------------------

Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None
{quote}

However, the next part where the scripts wait for the jobtracker to start time out:

Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist

      was (Author: znbailey):
    Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None

However, the next part where the scripts wait for the jobtracker to start time out:

Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913082#action_12913082 ] 

Zach Bailey commented on WHIRR-96:
----------------------------------

Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:

$ ls -la /dev/sd*

which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1).

I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.

> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:56 PM:
------------------------------------------------------------

Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{{Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None}}

However, the next part where the scripts wait for the jobtracker to start time out:

{{Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

{{Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist}}

      was (Author: znbailey):
    Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{{

Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None
}}

However, the next part where the scripts wait for the jobtracker to start time out:

{{
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

{{

ep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist
}}
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:59 PM:
------------------------------------------------------------

Contents of my cluster.cfg:

{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}

and my ec2-storage-dataclip-prod.json:

{quote}
\{
  "dn": [
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-923e17fb"
      }, 
      \{
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-943e17fd"
      }
    ], 
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-963e17ff"
      }, 
      \{
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-68311801"
      }
    ]
  ], 
  "nn": [
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-9a3e17f3"
      }, 
      \{
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-9c3e17f5"
      }
    ]
  ]
}
{quote}

      was (Author: znbailey):
    Contents of my cluster.cfg:

{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}

and my ec2-storage-dataclip-prod.json:

{quote}
{
  "dn": [
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-923e17fb"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-943e17fd"
      }
    ], 
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-963e17ff"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-68311801"
      }
    ]
  ], 
  "nn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-9a3e17f3"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-9c3e17f5"
      }
    ]
  ]
}
{quote}
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913082#action_12913082 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 1:07 PM:
-----------------------------------------------------------

Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:

$ ls -la /dev/sd*

which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1). And, obviously, without having any partition on the volume there's no possibility of them having a filesystem.

I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.

I followed the instructions here: 

http://help.rightscale.com/cgi-bin/rightscale.cfg/php/enduser/std_adp.php?p_faqid=12&p_created=1221775401

for partitioning and formatting my EBS volumes on each node. The only thing I did differently was using the ext3 filesystem instead of ext2.

      was (Author: znbailey):
    Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:

$ ls -la /dev/sd*

which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1).

I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ] 

Zach Bailey commented on WHIRR-96:
----------------------------------

Contents of my cluster.cfg:

{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}

and my ec2-storage-dataclip-prod.json:

{quote}
{
  "dn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-923e17fb"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-943e17fd"
      }
    ], 
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-963e17ff"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-68311801"
      }
    ]
  ], 
  "nn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-9a3e17f3"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-9c3e17f5"
      }
    ]
  ]
}
{quote}

> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:59 PM:
------------------------------------------------------------

Contents of my cluster.cfg:

{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}

and my ec2-storage-dataclip-prod.json:

{quote}
{
  "dn": [
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-923e17fb"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-943e17fd"
      }
    ], 
    [
      \{
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-963e17ff"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-68311801"
      }
    ]
  ], 
  "nn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-9a3e17f3"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-9c3e17f5"
      }
    ]
  ]
}
{quote}

      was (Author: znbailey):
    Contents of my cluster.cfg:

{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}

and my ec2-storage-dataclip-prod.json:

{quote}
{
  "dn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-923e17fb"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-943e17fd"
      }
    ], 
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-963e17ff"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-68311801"
      }
    ]
  ], 
  "nn": [
    [
      {
        "device": "/dev/sdj", 
        "mount_point": "/ebs1", 
        "volume_id": "vol-9a3e17f3"
      }, 
      {
        "device": "/dev/sdk", 
        "mount_point": "/ebs2", 
        "volume_id": "vol-9c3e17f5"
      }
    ]
  ]
}
{quote}
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:58 PM:
------------------------------------------------------------

Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None
{quote}

However, the next part where the scripts wait for the jobtracker to start time out:

{quote}
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
{quote}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

{quote}
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist
{quote}

      was (Author: znbailey):
    Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None
{quote}

However, the next part where the scripts wait for the jobtracker to start time out:

Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work with CDH3 on Lucid

Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ] 

Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:56 PM:
------------------------------------------------------------

Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None

However, the next part where the scripts wait for the jobtracker to start time out:

Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist

      was (Author: znbailey):
    Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:

{{Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started

Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started

Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn	vol-9a3e17f3	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
nn	vol-9c3e17f5	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:20.000Z	None
dn	vol-923e17fb	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:39.000Z	None
dn	vol-943e17fd	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:42.000Z	None
dn	vol-963e17ff	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:45.000Z	None
dn	vol-68311801	250	snap-76cafa1d	us-east-1a	in-use	2010-09-21T14:24:46.000Z	None}}

However, the next part where the scripts wait for the jobtracker to start time out:

{{Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}

Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:

{{Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [   88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [   98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [  108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [  118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [  128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [  138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [  148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [  158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [  168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [  178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [  188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [  198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [  208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [  218.409901] GFS2: gfs2 mount does not exist}}
  
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
>                 Key: WHIRR-96
>                 URL: https://issues.apache.org/jira/browse/WHIRR-96
>             Project: Whirr
>          Issue Type: Bug
>          Components: contrib/python
>            Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.