You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andras Barjak <an...@lynxanalytics.com> on 2014/06/03 19:05:07 UTC
mounting SSD devices of EC2 r3.8xlarge instances
Hi,
I have noticed that upon launching a cluster consisting of r3.8xlarge
high-memory instances the standard /mnt /mnt2 /mnt3 /mnt4 temporary
directories get created and set up for temp usage, however they will point
to the root 8Gb filesystem.
The 2x320GB SSD-s are not mounted and also they are not even formatted.
This problem might affect other EC2 instances as well, I suppose.
I am using 0.9.1, is this something that has been corrected in the 1.0.0
spark-ec2 script?
regards,
András Barják
Re: mounting SSD devices of EC2 r3.8xlarge instances
Posted by Han JU <ju...@gmail.com>.
For SSDs in r3, maybe it's better to mount with `discard` option since it
supports TRIM:
What I did for r3.large:
echo '/dev/xvdb /mnt ext4 defaults,noatime,nodiratime,discard 0 0' >>
/etc/fstab
mkfs.ext4 /dev/xvdb
mount /dev/xvdb
2014-06-03 19:15 GMT+02:00 Matei Zaharia <ma...@gmail.com>:
> Those instance types are not yet supported by the scripts, but
> https://issues.apache.org/jira/browse/SPARK-1790 is tracking this issue
> and it will soon be fixed in both branch-0.9 and 1.0. The problem is that
> those drives are not formatted on r3 machines, whereas they are on the
> other instance types, so the script assumed that they’d be formatted. You
> can manually go and do this to set them up:
>
> mkfs.ext4 /dev/sdb
> mkfs.ext4 /dev/sdc
> mount -o noatime /dev/sdb /mnt
> mount -o noatime /dev/sdc /mnt2
>
> Matei
>
> On Jun 3, 2014, at 10:05 AM, Andras Barjak <
> andras.barjak@lynxanalytics.com> wrote:
>
> > Hi,
> > I have noticed that upon launching a cluster consisting of r3.8xlarge
> high-memory instances the standard /mnt /mnt2 /mnt3 /mnt4 temporary
> directories get created and set up for temp usage, however they will point
> to the root 8Gb filesystem.
> > The 2x320GB SSD-s are not mounted and also they are not even formatted.
> >
> > This problem might affect other EC2 instances as well, I suppose.
> > I am using 0.9.1, is this something that has been corrected in the 1.0.0
> spark-ec2 script?
> >
> > regards,
> > András Barják
>
>
--
*JU Han*
Data Engineer @ Botify.com
+33 0619608888
Re: mounting SSD devices of EC2 r3.8xlarge instances
Posted by Matei Zaharia <ma...@gmail.com>.
Those instance types are not yet supported by the scripts, but https://issues.apache.org/jira/browse/SPARK-1790 is tracking this issue and it will soon be fixed in both branch-0.9 and 1.0. The problem is that those drives are not formatted on r3 machines, whereas they are on the other instance types, so the script assumed that they’d be formatted. You can manually go and do this to set them up:
mkfs.ext4 /dev/sdb
mkfs.ext4 /dev/sdc
mount -o noatime /dev/sdb /mnt
mount -o noatime /dev/sdc /mnt2
Matei
On Jun 3, 2014, at 10:05 AM, Andras Barjak <an...@lynxanalytics.com> wrote:
> Hi,
> I have noticed that upon launching a cluster consisting of r3.8xlarge high-memory instances the standard /mnt /mnt2 /mnt3 /mnt4 temporary directories get created and set up for temp usage, however they will point to the root 8Gb filesystem.
> The 2x320GB SSD-s are not mounted and also they are not even formatted.
>
> This problem might affect other EC2 instances as well, I suppose.
> I am using 0.9.1, is this something that has been corrected in the 1.0.0 spark-ec2 script?
>
> regards,
> András Barják