You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Tushar Sudake <et...@gmail.com> on 2017/10/09 01:54:58 UTC

EMR: Use extra mounted EBS volumes for spark.local.dir

Hello everyone,

I'm using 'r4.8xlarge' instances on EMR for my Spark Application.
To each node, I'm attaching one 512 GB EBS volume.

By logging in into nodes I tried verifying that this volume is being set
for 'spark.local.dir' by EMR automatically, but couldn't find any such
configuration.

Can someone please confirm this? Do I need to do it myself though bootstrap
steps?

Thanks,
Tushar

Re: EMR: Use extra mounted EBS volumes for spark.local.dir

Posted by Vadim Semenov <va...@datadoghq.com>.

that's probably better be directed to the AWS support

On Sun, Oct 8, 2017 at 9:54 PM, Tushar Sudake <et...@gmail.com> wrote:

> Hello everyone,
>
> I'm using 'r4.8xlarge' instances on EMR for my Spark Application.
> To each node, I'm attaching one 512 GB EBS volume.
>
> By logging in into nodes I tried verifying that this volume is being set
> for 'spark.local.dir' by EMR automatically, but couldn't find any such
> configuration.
>
> Can someone please confirm this? Do I need to do it myself though
> bootstrap steps?
>
> Thanks,
> Tushar
>