You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2010/09/21 18:49:32 UTC
[jira] Created: (WHIRR-96) EBS cluster doesn't work with CDH3 on
Lucid
EBS cluster doesn't work with CDH3 on Lucid
-------------------------------------------
Key: WHIRR-96
URL: https://issues.apache.org/jira/browse/WHIRR-96
Project: Whirr
Issue Type: Bug
Components: contrib/python
Reporter: Tom White
Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on
Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ]
Zach Bailey commented on WHIRR-96:
----------------------------------
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{{
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
}}
However, the next part where the scripts wait for the jobtracker to start time out:
{{
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
{{
ep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
}}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:57 PM:
------------------------------------------------------------
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
{quote}
However, the next part where the scripts wait for the jobtracker to start time out:
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
was (Author: znbailey):
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
However, the next part where the scripts wait for the jobtracker to start time out:
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on
Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913082#action_12913082 ]
Zach Bailey commented on WHIRR-96:
----------------------------------
Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:
$ ls -la /dev/sd*
which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1).
I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:56 PM:
------------------------------------------------------------
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{{Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None}}
However, the next part where the scripts wait for the jobtracker to start time out:
{{Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
{{Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist}}
was (Author: znbailey):
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{{
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
}}
However, the next part where the scripts wait for the jobtracker to start time out:
{{
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
{{
ep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
}}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:59 PM:
------------------------------------------------------------
Contents of my cluster.cfg:
{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}
and my ec2-storage-dataclip-prod.json:
{quote}
\{
"dn": [
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-923e17fb"
},
\{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-943e17fd"
}
],
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-963e17ff"
},
\{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-68311801"
}
]
],
"nn": [
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-9a3e17f3"
},
\{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-9c3e17f5"
}
]
]
}
{quote}
was (Author: znbailey):
Contents of my cluster.cfg:
{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}
and my ec2-storage-dataclip-prod.json:
{quote}
{
"dn": [
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-923e17fb"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-943e17fd"
}
],
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-963e17ff"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-68311801"
}
]
],
"nn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-9a3e17f3"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-9c3e17f5"
}
]
]
}
{quote}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913082#action_12913082 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 1:07 PM:
-----------------------------------------------------------
Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:
$ ls -la /dev/sd*
which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1). And, obviously, without having any partition on the volume there's no possibility of them having a filesystem.
I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.
I followed the instructions here:
http://help.rightscale.com/cgi-bin/rightscale.cfg/php/enduser/std_adp.php?p_faqid=12&p_created=1221775401
for partitioning and formatting my EBS volumes on each node. The only thing I did differently was using the ext3 filesystem instead of ext2.
was (Author: znbailey):
Another problem I ran into initially was the volumes created using the "create-storage" command were neither partitioned nor formatted with a filesystem. I verified this by logging on to the nodes individually and running:
$ ls -la /dev/sd*
which returned only the raw devices (/dev/sdj and /dev/sdk). Had those volumes been partitioned I would have at least seen two more entries (/dev/sdj1 and /dev/sdk1).
I thought this was the root cause of the problem that was keeping the jobtracker from starting but ultimately even after fixing this problem the volumes still fail to mount cleanly.
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WHIRR-96) EBS cluster doesn't work with CDH3 on
Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ]
Zach Bailey commented on WHIRR-96:
----------------------------------
Contents of my cluster.cfg:
{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}
and my ec2-storage-dataclip-prod.json:
{quote}
{
"dn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-923e17fb"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-943e17fd"
}
],
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-963e17ff"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-68311801"
}
]
],
"nn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-9a3e17f3"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-9c3e17f5"
}
]
]
}
{quote}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913077#action_12913077 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:59 PM:
------------------------------------------------------------
Contents of my cluster.cfg:
{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}
and my ec2-storage-dataclip-prod.json:
{quote}
{
"dn": [
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-923e17fb"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-943e17fd"
}
],
[
\{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-963e17ff"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-68311801"
}
]
],
"nn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-9a3e17f3"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-9c3e17f5"
}
]
]
}
{quote}
was (Author: znbailey):
Contents of my cluster.cfg:
{quote}
[dataclip-prod]
image_id=ami-2d4aa444
instance_type=c1.medium
key_name=zach.bailey
availability_zone=us-east-1a
private_key=/Users/zach/Downloads/zachbailey.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
user_data_file=http://archive.cloudera.com/cloud/ec2/cdh3/hadoop-ec2-init-remote.sh
{quote}
and my ec2-storage-dataclip-prod.json:
{quote}
{
"dn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-923e17fb"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-943e17fd"
}
],
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-963e17ff"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-68311801"
}
]
],
"nn": [
[
{
"device": "/dev/sdj",
"mount_point": "/ebs1",
"volume_id": "vol-9a3e17f3"
},
{
"device": "/dev/sdk",
"mount_point": "/ebs2",
"volume_id": "vol-9c3e17f5"
}
]
]
}
{quote}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:58 PM:
------------------------------------------------------------
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
{quote}
However, the next part where the scripts wait for the jobtracker to start time out:
{quote}
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
{quote}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
{quote}
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
{quote}
was (Author: znbailey):
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{quote}
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
{quote}
However, the next part where the scripts wait for the jobtracker to start time out:
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (WHIRR-96) EBS cluster doesn't work
with CDH3 on Lucid
Posted by "Zach Bailey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-96?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913076#action_12913076 ]
Zach Bailey edited comment on WHIRR-96 at 9/21/10 12:56 PM:
------------------------------------------------------------
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None
However, the next part where the scripts wait for the jobtracker to start time out:
Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist
was (Author: znbailey):
Followed the instructions on the page above to the letter besides using a different cluster name. When starting the cluster, the instances start correctly and the volumes are attached fine:
{{Waiting for 1 instances in role nn,snn,jt to start
.....................................nn,snn,jt instances started
Waiting for 2 instances in role dn,tt to start
.......................................dn,tt instances started
Waiting 10 seconds before attaching storage
Attaching storage to i-eb5a6581
Attaching vol-9a3e17f3 to i-eb5a6581
Attaching vol-9c3e17f5 to i-eb5a6581
Attaching storage to i-df5a65b5
Attaching vol-923e17fb to i-df5a65b5
Attaching vol-943e17fd to i-df5a65b5
Attaching storage to i-dd5a65b7
Attaching vol-963e17ff to i-dd5a65b7
Attaching vol-68311801 to i-dd5a65b7
nn vol-9a3e17f3 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
nn vol-9c3e17f5 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:20.000Z None
dn vol-923e17fb 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:39.000Z None
dn vol-943e17fd 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:42.000Z None
dn vol-963e17ff 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:45.000Z None
dn vol-68311801 250 snap-76cafa1d us-east-1a in-use 2010-09-21T14:24:46.000Z None}}
However, the next part where the scripts wait for the jobtracker to start time out:
{{Waiting for jobtracker to start
.....................................................................................................................................................................................................................................................................................................Timeout while waiting for Hadoop to start. Please check logs on cluster.}}
Checking /var/log/messages on the cluster NN yield the following error spamming the logs approx every 10 seconds:
{{Sep 21 16:35:59 domU-12-31-39-06-18-D1 ec2: #############################################################
Sep 21 16:37:21 domU-12-31-39-06-18-D1 kernel: [ 88.139988] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:31 domU-12-31-39-06-18-D1 kernel: [ 98.145469] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:41 domU-12-31-39-06-18-D1 kernel: [ 108.151421] GFS2: path_lookup on /dev/sdk returned error -2
Sep 21 16:37:51 domU-12-31-39-06-18-D1 kernel: [ 118.178490] GFS2: gfs2 mount does not exist
Sep 21 16:38:02 domU-12-31-39-06-18-D1 kernel: [ 128.205265] GFS2: gfs2 mount does not exist
Sep 21 16:38:12 domU-12-31-39-06-18-D1 kernel: [ 138.228104] GFS2: gfs2 mount does not exist
Sep 21 16:38:22 domU-12-31-39-06-18-D1 kernel: [ 148.252729] GFS2: gfs2 mount does not exist
Sep 21 16:38:32 domU-12-31-39-06-18-D1 kernel: [ 158.277586] GFS2: gfs2 mount does not exist
Sep 21 16:38:42 domU-12-31-39-06-18-D1 kernel: [ 168.311393] GFS2: gfs2 mount does not exist
Sep 21 16:38:52 domU-12-31-39-06-18-D1 kernel: [ 178.329587] GFS2: gfs2 mount does not exist
Sep 21 16:39:02 domU-12-31-39-06-18-D1 kernel: [ 188.349149] GFS2: gfs2 mount does not exist
Sep 21 16:39:12 domU-12-31-39-06-18-D1 kernel: [ 198.369786] GFS2: gfs2 mount does not exist
Sep 21 16:39:22 domU-12-31-39-06-18-D1 kernel: [ 208.389564] GFS2: gfs2 mount does not exist
Sep 21 16:39:32 domU-12-31-39-06-18-D1 kernel: [ 218.409901] GFS2: gfs2 mount does not exist}}
> EBS cluster doesn't work with CDH3 on Lucid
> -------------------------------------------
>
> Key: WHIRR-96
> URL: https://issues.apache.org/jira/browse/WHIRR-96
> Project: Whirr
> Issue Type: Bug
> Components: contrib/python
> Reporter: Tom White
>
> Following the instructions with https://docs.cloudera.com/display/DOC/Using+Persistent+Clusters and ami-2d4aa444 gives "Timeout while waiting for Hadoop to start. Please check logs on cluster."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.