best practice to use host local drive?

[Joe Stein <jo...@stealth.ly>]

Hi, I wanted to get recommends on what is the best practice within a
launched task by Aurora for accessing some local drive (i.e.
/mnt/data/stuff) within the process running.

Thanks!

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

Re: best practice to use host local drive?

[Chris Lambert <cl...@twitter.com.INVALID>]

On Tue, Aug 12, 2014 at 1:18 PM, Bill Farner <wf...@apache.org> wrote:

> The answer will depend on the use case.  Here's the three most common:
>
> - read/write scratch space (e.g. /tmp)
>
> Prefer using the sandbox instead (avoid the problem).  Otherwise, create a
> directory containing a UUID and try to clean up after yourself.  This is
> ultimately brittle.
>

Just FYI:  there's a Mesos ticket for (optional) per-container tmp dirs,
including an outstanding review:
https://issues.apache.org/jira/browse/MESOS-1586


> - read-only data available on every host
>
> No big deal (aside for races when the data changes), read away.
>
> - read/write, persistent data
>
> There's not a great story here, at least not in the traditional
> mesos/aurora "run anywhere" model.  The best approach we have so far is to
> mount to a directory owned by the user that will access the data, constrain
> tasks to run on that host (dedicated=dba/mysql), and configure the slave
> with the dedicated attribute having the same value.
>
>
>
>
>
> -=Bill
>
>
> On Tue, Aug 12, 2014 at 1:00 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Hi, I wanted to get recommends on what is the best practice within a
> > launched task by Aurora for accessing some local drive (i.e.
> > /mnt/data/stuff) within the process running.
> >
> > Thanks!
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>

Re: best practice to use host local drive?

[Jay Buffington <me...@jaybuff.com>]

On Tue, Aug 12, 2014 at 1:18 PM, Bill Farner <wf...@apache.org> wrote:
> - read/write, persistent data
>
> There's not a great story here, at least not in the traditional
> mesos/aurora "run anywhere" model.  The best approach we have so far is to
> mount to a directory owned by the user that will access the data, constrain
> tasks to run on that host (dedicated=dba/mysql), and configure the slave
> with the dedicated attribute having the same value.

There is a proposal in mesos for the concept of persistent resources.  See
https://issues.apache.org/jira/browse/MESOS-1554

If this was implemented it would give Aurora the ability to launch a task that
used the same storage as a previous task used.

Jay

Re: best practice to use host local drive?

[Bill Farner <wf...@apache.org>]

The answer will depend on the use case.  Here's the three most common:

- read/write scratch space (e.g. /tmp)

Prefer using the sandbox instead (avoid the problem).  Otherwise, create a
directory containing a UUID and try to clean up after yourself.  This is
ultimately brittle.

- read-only data available on every host

No big deal (aside for races when the data changes), read away.

- read/write, persistent data

There's not a great story here, at least not in the traditional
mesos/aurora "run anywhere" model.  The best approach we have so far is to
mount to a directory owned by the user that will access the data, constrain
tasks to run on that host (dedicated=dba/mysql), and configure the slave
with the dedicated attribute having the same value.





-=Bill


On Tue, Aug 12, 2014 at 1:00 PM, Joe Stein <jo...@stealth.ly> wrote:

> Hi, I wanted to get recommends on what is the best practice within a
> launched task by Aurora for accessing some local drive (i.e.
> /mnt/data/stuff) within the process running.
>
> Thanks!
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>