You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Arunabha Ghosh <ar...@gmail.com> on 2014/10/08 01:06:22 UTC

Multiple disks with Mesos

Hi,
     I would like to run Mesos slaves on machines that have multiple disks.
According to the Mesos configuration page
<http://mesos.apache.org/documentation/latest/configuration/> I can specify
a work_dir argument to the slaves.

1) Can the work_dir argument contain multiple directories ?

2) Is the work_dir where Mesos will place all of its data ? So If I started
a task on Mesos, would the slave place the task's data (stderr, stdout,
task created directories) inside work_dir ?

Thanks,
Arunabha

Re: Multiple disks with Mesos

Posted by Dick Davies <di...@hellooperator.net>.
To answer point 2) - yes, your executors will create their 'sandboxes'
under work_dir.


On 8 October 2014 00:13, Arunabha Ghosh <ar...@gmail.com> wrote:
> Thanks Steven !
>
> On Tue, Oct 7, 2014 at 4:08 PM, Steven Schlansker
> <ss...@opentable.com> wrote:
>>
>>
>> On Oct 7, 2014, at 4:06 PM, Arunabha Ghosh <ar...@gmail.com> wrote:
>>
>> > Hi,
>> >      I would like to run Mesos slaves on machines that have multiple
>> > disks. According to the Mesos configuration page I can specify a work_dir
>> > argument to the slaves.
>> >
>> > 1) Can the work_dir argument contain multiple directories ?
>> >
>> > 2) Is the work_dir where Mesos will place all of its data ? So If I
>> > started a task on Mesos, would the slave place the task's data (stderr,
>> > stdout, task created directories) inside work_dir ?
>>
>> We stitch our disks together before Mesos gets its hands on it using a
>> technology such as LVM or btrfs, so that the work_dir is spread across the
>> multiple disks transparently.
>>
>

Re: Multiple disks with Mesos

Posted by Arunabha Ghosh <ar...@gmail.com>.
Thanks Steven !

On Tue, Oct 7, 2014 at 4:08 PM, Steven Schlansker <sschlansker@opentable.com
> wrote:

>
> On Oct 7, 2014, at 4:06 PM, Arunabha Ghosh <ar...@gmail.com> wrote:
>
> > Hi,
> >      I would like to run Mesos slaves on machines that have multiple
> disks. According to the Mesos configuration page I can specify a work_dir
> argument to the slaves.
> >
> > 1) Can the work_dir argument contain multiple directories ?
> >
> > 2) Is the work_dir where Mesos will place all of its data ? So If I
> started a task on Mesos, would the slave place the task's data (stderr,
> stdout, task created directories) inside work_dir ?
>
> We stitch our disks together before Mesos gets its hands on it using a
> technology such as LVM or btrfs, so that the work_dir is spread across the
> multiple disks transparently.
>
>

Re: Multiple disks with Mesos

Posted by Tim St Clair <ts...@redhat.com>.
+1, stitching can be done outside of mesos prior to init. 

----- Original Message -----
> From: "Steven Schlansker" <ss...@opentable.com>
> To: user@mesos.apache.org
> Sent: Tuesday, October 7, 2014 6:08:21 PM
> Subject: Re: Multiple disks with Mesos
> 
> 
> On Oct 7, 2014, at 4:06 PM, Arunabha Ghosh <ar...@gmail.com> wrote:
> 
> > Hi,
> >      I would like to run Mesos slaves on machines that have multiple disks.
> >      According to the Mesos configuration page I can specify a work_dir
> >      argument to the slaves.
> > 
> > 1) Can the work_dir argument contain multiple directories ?
> > 
> > 2) Is the work_dir where Mesos will place all of its data ? So If I started
> > a task on Mesos, would the slave place the task's data (stderr, stdout,
> > task created directories) inside work_dir ?
> 
> We stitch our disks together before Mesos gets its hands on it using a
> technology such as LVM or btrfs, so that the work_dir is spread across the
> multiple disks transparently.
> 
> 

-- 
Cheers,
Timothy St. Clair
Red Hat Inc.

Re: Multiple disks with Mesos

Posted by Steven Schlansker <ss...@opentable.com>.
On Oct 7, 2014, at 4:06 PM, Arunabha Ghosh <ar...@gmail.com> wrote:

> Hi,
>      I would like to run Mesos slaves on machines that have multiple disks. According to the Mesos configuration page I can specify a work_dir argument to the slaves. 
> 
> 1) Can the work_dir argument contain multiple directories ?
> 
> 2) Is the work_dir where Mesos will place all of its data ? So If I started a task on Mesos, would the slave place the task's data (stderr, stdout, task created directories) inside work_dir ?

We stitch our disks together before Mesos gets its hands on it using a technology such as LVM or btrfs, so that the work_dir is spread across the multiple disks transparently.


Re: Multiple disks with Mesos

Posted by Damien Hardy <dh...@viadeoteam.com>.
Hello,

I run mesos on top hadoop HDFS.
Hadoop handle well with JBOD configuration.

Today mesos can only work on one of the disk and cannot take advantage
of other disks. (use non HDFS space)

This would be a great feature to handle with JBOD too. Dealing with
failure better than LVM for example.

Cheers,

Le 08/10/2014 01:06, Arunabha Ghosh a écrit :
> Hi,
>      I would like to run Mesos slaves on machines that have multiple
> disks. According to the Mesos configuration page
> <http://mesos.apache.org/documentation/latest/configuration/> I can
> specify a work_dir argument to the slaves. 
> 
> 1) Can the work_dir argument contain multiple directories ?
> 
> 2) Is the work_dir where Mesos will place all of its data ? So If I
> started a task on Mesos, would the slave place the task's data (stderr,
> stdout, task created directories) inside work_dir ?
> 
> Thanks,
> Arunabha

-- 
Damien HARDY