You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/12/30 20:37:29 UTC
[jira] Created: (HBASE-2080) [EC2] Support multivolume local
instance storage
[EC2] Support multivolume local instance storage
------------------------------------------------
Key: HBASE-2080
URL: https://issues.apache.org/jira/browse/HBASE-2080
Project: Hadoop HBase
Issue Type: Sub-task
Reporter: Andrew Purtell
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2080) [EC2] Support multivolume local
instance storage
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2080:
----------------------------------
Attachment: HBASE-2080.patch
This is the relevant hunk:
{noformat}
+# Probe for instance volumes
+
+# /dev/sdb as /mnt is always set up by base image
+DFS_NAME_DIR="/mnt/hadoop/dfs/name"
+DFS_DATA_DIR="/mnt/hadoop/dfs/data"
+i=2
+for d in c d e f g h i j k l m n o p q r s t u v w x y z; do
+ m="/mnt${i}"
+ mkdir -p $m
+ mount /dev/sd${d} $m > /dev/null 2>&1
+ if [ $? -eq 0 ] ; then
+ if [ $i -lt 3 ] ; then # no more than two namedirs
+ DFS_NAME_DIR="${DFS_NAME_DIR},${m}/hadoop/dfs/name"
+ fi
+ DFS_DATA_DIR="${DFS_DATA_DIR},${m}/hadoop/dfs/data"
+ i=$(( i + 1 ))
+ fi
+done
{noformat}
> [EC2] Support multivolume local instance storage
> ------------------------------------------------
>
> Key: HBASE-2080
> URL: https://issues.apache.org/jira/browse/HBASE-2080
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2080.patch
>
>
> The (old) Hadoop EC2 scripts we used as a basis for our scripts do not properly handle instance types that have their local storage attached as multiple volumes. The AMI build script uses Amazon's latest Fedora 8 AMI as a base, which does not automatically mount instance storage on multiple volumes either.
> We recommend use of two instance types:
> - c1.medium for zookeeper, which has one vdisk of 340 GB as /dev/sdb mounted on /mnt by the base image
> - c1.xlarge for master and slaves, which have four vdisks of 420 GB as /dev/sd[bcde], only one of which is mounted on /mnt by the base image
> Additionally, the m1.xlarge instance type, which a user might use anyway, has two vdisks of 420 GB as /dev/sd[bc], only one of which is mounted on /mnt by the base image.
> The hbase-ec2-init-remote.sh script should probe for all available instance storage devices, mount them, and update the DataNode configuration appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2080) [EC2] Support multivolume local
instance storage
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-2080.
-----------------------------------
Resolution: Fixed
> [EC2] Support multivolume local instance storage
> ------------------------------------------------
>
> Key: HBASE-2080
> URL: https://issues.apache.org/jira/browse/HBASE-2080
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.3, 0.21.0
>
> Attachments: HBASE-2080.patch
>
>
> The (old) Hadoop EC2 scripts we used as a basis for our scripts do not properly handle instance types that have their local storage attached as multiple volumes. The AMI build script uses Amazon's latest Fedora 8 AMI as a base, which does not automatically mount instance storage on multiple volumes either.
> We recommend use of two instance types:
> - c1.medium for zookeeper, which has one vdisk of 340 GB as /dev/sdb mounted on /mnt by the base image
> - c1.xlarge for master and slaves, which have four vdisks of 420 GB as /dev/sd[bcde], only one of which is mounted on /mnt by the base image
> Additionally, the m1.xlarge instance type, which a user might use anyway, has two vdisks of 420 GB as /dev/sd[bc], only one of which is mounted on /mnt by the base image.
> The hbase-ec2-init-remote.sh script should probe for all available instance storage devices, mount them, and update the DataNode configuration appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2080) [EC2] Support multivolume local
instance storage
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2080:
----------------------------------
Fix Version/s: 0.21.0
> [EC2] Support multivolume local instance storage
> ------------------------------------------------
>
> Key: HBASE-2080
> URL: https://issues.apache.org/jira/browse/HBASE-2080
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.3, 0.21.0
>
>
> The (old) Hadoop EC2 scripts we used as a basis for our scripts do not properly handle instance types that have their local storage attached as multiple volumes. The AMI build script uses Amazon's latest Fedora 8 AMI as a base, which does not automatically mount instance storage on multiple volumes either.
> We recommend use of two instance types:
> - c1.medium for zookeeper, which has one vdisk of 340 GB as /dev/sdb mounted on /mnt by the base image
> - c1.xlarge for master and slaves, which have four vdisks of 420 GB as /dev/sd[bcde], only one of which is mounted on /mnt by the base image
> Additionally, the m1.xlarge instance type, which a user might use anyway, has two vdisks of 420 GB as /dev/sd[bc], only one of which is mounted on /mnt by the base image.
> The hbase-ec2-init-remote.sh script should probe for all available instance storage devices, mount them, and update the DataNode configuration appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2080) [EC2] Support multivolume local
instance storage
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-2080:
----------------------------------
Description:
The (old) Hadoop EC2 scripts we used as a basis for our scripts do not properly handle instance types that have their local storage attached as multiple volumes. The AMI build script uses Amazon's latest Fedora 8 AMI as a base, which does not automatically mount instance storage on multiple volumes either.
We recommend use of two instance types:
- c1.medium for zookeeper, which has one vdisk of 340 GB as /dev/sdb mounted on /mnt by the base image
- c1.xlarge for master and slaves, which have four vdisks of 420 GB as /dev/sd[bcde], only one of which is mounted on /mnt by the base image
Additionally, the m1.xlarge instance type, which a user might use anyway, has two vdisks of 420 GB as /dev/sd[bc], only one of which is mounted on /mnt by the base image.
The hbase-ec2-init-remote.sh script should probe for all available instance storage devices, mount them, and update the DataNode configuration appropriately.
Fix Version/s: 0.20.3
Assignee: Andrew Purtell
> [EC2] Support multivolume local instance storage
> ------------------------------------------------
>
> Key: HBASE-2080
> URL: https://issues.apache.org/jira/browse/HBASE-2080
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.3, 0.21.0
>
>
> The (old) Hadoop EC2 scripts we used as a basis for our scripts do not properly handle instance types that have their local storage attached as multiple volumes. The AMI build script uses Amazon's latest Fedora 8 AMI as a base, which does not automatically mount instance storage on multiple volumes either.
> We recommend use of two instance types:
> - c1.medium for zookeeper, which has one vdisk of 340 GB as /dev/sdb mounted on /mnt by the base image
> - c1.xlarge for master and slaves, which have four vdisks of 420 GB as /dev/sd[bcde], only one of which is mounted on /mnt by the base image
> Additionally, the m1.xlarge instance type, which a user might use anyway, has two vdisks of 420 GB as /dev/sd[bc], only one of which is mounted on /mnt by the base image.
> The hbase-ec2-init-remote.sh script should probe for all available instance storage devices, mount them, and update the DataNode configuration appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.