You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by james <ga...@verizon.net> on 2016/05/11 02:09:06 UTC
distributed file systems
Hello,
Has anyone customized/compiled mesos and successfully deployed a mesos
cluster to run on cephfs, orangefs [1], or any other distributed file
systems?
If so, some detail on your setup would be appreciated.
[1]
http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6
RE: distributed file systems
Posted by Aaron Carey <ac...@ilm.com>.
What workloads are you attempting to run exactly?
Is this for an actual implementation or a research project?
Thanks,
Aaron
--
Aaron Carey
Production Engineer - Cloud Pipeline
Industrial Light & Magic
London
020 3751 9150
________________________________________
From: james [garftd@verizon.net]
Sent: 11 May 2016 17:46
To: user@mesos.apache.org
Subject: Re: distributed file systems
On 05/11/2016 10:09 AM, Aaron Carey wrote:
> What exactly do you mean by deploying a mesos cluster to run on ceph etc?
>
> Do you mean having a clustered file system mounted via nfs to the hosts which contains the
> mesos binaries?
That would be one way to use a DFS, but low latency on a variety of
different linux kernels will be evaluated too. So will the testing of
ulibc-ng, musl and other fundamental components, as to the trade offs
of performance vs robustness of supported frameworks.
> Or something to do with how jobs are executed?
YES, exactly. In fact, it is most interesting to test with Hi
Performance Computing (HPC) on top of Mesos. The simple way to think
about one of the ultimate goals of the research is how to completely
replace Hadoop with Mesos, a DFS and many other components. Portability,
not limited to Arm64v8 is a companion track related to HPC on clusters
and bare metal hardware. But most HPC installations moving forward, will
have to also support some typical (admin/web/security/AI/etc) types of
workloads, as part of the evaluation.
Mesos does seem to be flexible enough for these sorts of explorations
and experiments. There is no inclination for constricting ideas and
testing of competing components; only the desire to experiment, share
and refine the component mix (aka the cluster-stack), all in open
forums. I. E. Cluster-Stack_A vs Cluster-Stack_B etc etc for various
workloads, on fixed, modest cluster sizes.
Think of it as experimentation that will eventually lead to published
results, such as what mips/mops and various benchmarks have done for
hardware, only the cluster-stack is the variable(s) under study.
All ideas, comments and criticisms are most welcome. HDFS is mostly
described as the main bottleneck to HPC efforts and much of what folks
are doing with various DFS replacements is being too strictly
constrained, imho.
James
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
> ________________________________________
> From: james [garftd@verizon.net]
> Sent: 11 May 2016 17:08
> To: user@mesos.apache.org
> Subject: Re: distributed file systems
>
> Hello Rodrick,
>
> That EFS looks interesting, but I did not find the location for the
> source-code/git download? I do not remember the (linux) kernel hooks
> for that Distributed File System, or is completely on top of the systems
> codes?
>
> License details and I'm not sure if it's 100% opensource?
>
> Beegfs [A] is partially opensource, but that does not fit what is needed
> for experimentation. A robust community around open sources and tools,
> such as github, should have been mentioned. Equally important
> is a community keen on sharing and supporting other efforts to replicate
> and use the components of these cluster centric codes. [B,C]
>
>
> James
>
> [A] http://www.beegfs.com/content/
>
> [B] https://forums.aws.amazon.com/thread.jspa?threadID=217783
>
> [C]
> http://searchaws.techtarget.com/news/4500272286/Amazon-EFS-stuck-in-beta-lacks-native-Windows-support
>
>
>
>
> On 05/11/2016 01:07 AM, Rodrick Brown wrote:
>> Does EFS count? :-)
>>
>> https://aws.amazon.com/efs/
>>
>>
>> --
>>
>> *Rodrick Brown* / Systems Engineer
>>
>> +1 917 445 6839 / rodrick@orchardplatform.com
>> <ma...@orchardplatform.com>
>>
>> *Orchard Platform*
>>
>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>
>> http://www.orchardplatform.com <http://www.orchardplatform.com/>
>>
>> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
>> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>>
>>
>> On May 10 2016, at 9:07 pm, james <ga...@verizon.net> wrote:
>>
>> Hello,
>>
>>
>> Has anyone customized/compiled mesos and successfully deployed a mesos
>> cluster to run on cephfs, orangefs [1], or any other distributed file
>> systems?
>>
>> If so, some detail on your setup would be appreciated.
>>
>>
>> [1]
>> http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6
>>
>>
>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
>> for the use of the addressee only. If you are not an intended recipient
>> of this communication, please delete it immediately and notify the
>> sender by return email. Unauthorized reading, dissemination,
>> distribution or copying of this communication is prohibited. This
>> communication does not constitute an offer to sell or a solicitation of
>> an indication of interest to purchase any loan, security or any other
>> financial product or instrument, nor is it an offer to sell or a
>> solicitation of an indication of interest to purchase any products or
>> services to any persons who are prohibited from receiving such
>> information under applicable law. The contents of this communication may
>> not be accurate or complete and are subject to change without notice. As
>> such, Orchard App, Inc. (including its subsidiaries and affiliates,
>> "Orchard") makes no representation regarding the accuracy or
>> completeness of the information contained herein. The intended recipient
>> is advised to consult its own professional advisors, including those
>> specializing in legal, tax and accounting matters. Orchard does not
>> provide legal, tax or accounting advice.
>
>
Re: distributed file systems
Posted by james <ga...@verizon.net>.
On 05/11/2016 10:09 AM, Aaron Carey wrote:
> What exactly do you mean by deploying a mesos cluster to run on ceph etc?
>
> Do you mean having a clustered file system mounted via nfs to the hosts which contains the
> mesos binaries?
That would be one way to use a DFS, but low latency on a variety of
different linux kernels will be evaluated too. So will the testing of
ulibc-ng, musl and other fundamental components, as to the trade offs
of performance vs robustness of supported frameworks.
> Or something to do with how jobs are executed?
YES, exactly. In fact, it is most interesting to test with Hi
Performance Computing (HPC) on top of Mesos. The simple way to think
about one of the ultimate goals of the research is how to completely
replace Hadoop with Mesos, a DFS and many other components. Portability,
not limited to Arm64v8 is a companion track related to HPC on clusters
and bare metal hardware. But most HPC installations moving forward, will
have to also support some typical (admin/web/security/AI/etc) types of
workloads, as part of the evaluation.
Mesos does seem to be flexible enough for these sorts of explorations
and experiments. There is no inclination for constricting ideas and
testing of competing components; only the desire to experiment, share
and refine the component mix (aka the cluster-stack), all in open
forums. I. E. Cluster-Stack_A vs Cluster-Stack_B etc etc for various
workloads, on fixed, modest cluster sizes.
Think of it as experimentation that will eventually lead to published
results, such as what mips/mops and various benchmarks have done for
hardware, only the cluster-stack is the variable(s) under study.
All ideas, comments and criticisms are most welcome. HDFS is mostly
described as the main bottleneck to HPC efforts and much of what folks
are doing with various DFS replacements is being too strictly
constrained, imho.
James
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
> ________________________________________
> From: james [garftd@verizon.net]
> Sent: 11 May 2016 17:08
> To: user@mesos.apache.org
> Subject: Re: distributed file systems
>
> Hello Rodrick,
>
> That EFS looks interesting, but I did not find the location for the
> source-code/git download? I do not remember the (linux) kernel hooks
> for that Distributed File System, or is completely on top of the systems
> codes?
>
> License details and I'm not sure if it's 100% opensource?
>
> Beegfs [A] is partially opensource, but that does not fit what is needed
> for experimentation. A robust community around open sources and tools,
> such as github, should have been mentioned. Equally important
> is a community keen on sharing and supporting other efforts to replicate
> and use the components of these cluster centric codes. [B,C]
>
>
> James
>
> [A] http://www.beegfs.com/content/
>
> [B] https://forums.aws.amazon.com/thread.jspa?threadID=217783
>
> [C]
> http://searchaws.techtarget.com/news/4500272286/Amazon-EFS-stuck-in-beta-lacks-native-Windows-support
>
>
>
>
> On 05/11/2016 01:07 AM, Rodrick Brown wrote:
>> Does EFS count? :-)
>>
>> https://aws.amazon.com/efs/
>>
>>
>> --
>>
>> *Rodrick Brown* / Systems Engineer
>>
>> +1 917 445 6839 / rodrick@orchardplatform.com
>> <ma...@orchardplatform.com>
>>
>> *Orchard Platform*
>>
>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>
>> http://www.orchardplatform.com <http://www.orchardplatform.com/>
>>
>> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
>> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>>
>>
>> On May 10 2016, at 9:07 pm, james <ga...@verizon.net> wrote:
>>
>> Hello,
>>
>>
>> Has anyone customized/compiled mesos and successfully deployed a mesos
>> cluster to run on cephfs, orangefs [1], or any other distributed file
>> systems?
>>
>> If so, some detail on your setup would be appreciated.
>>
>>
>> [1]
>> http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6
>>
>>
>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
>> for the use of the addressee only. If you are not an intended recipient
>> of this communication, please delete it immediately and notify the
>> sender by return email. Unauthorized reading, dissemination,
>> distribution or copying of this communication is prohibited. This
>> communication does not constitute an offer to sell or a solicitation of
>> an indication of interest to purchase any loan, security or any other
>> financial product or instrument, nor is it an offer to sell or a
>> solicitation of an indication of interest to purchase any products or
>> services to any persons who are prohibited from receiving such
>> information under applicable law. The contents of this communication may
>> not be accurate or complete and are subject to change without notice. As
>> such, Orchard App, Inc. (including its subsidiaries and affiliates,
>> "Orchard") makes no representation regarding the accuracy or
>> completeness of the information contained herein. The intended recipient
>> is advised to consult its own professional advisors, including those
>> specializing in legal, tax and accounting matters. Orchard does not
>> provide legal, tax or accounting advice.
>
>
RE: distributed file systems
Posted by Aaron Carey <ac...@ilm.com>.
What exactly do you mean by deploying a mesos cluster to run on ceph etc?
Do you mean having a clustered file system mounted via nfs to the hosts which contains the mesos binaries?
Or something to do with how jobs are executed?
--
Aaron Carey
Production Engineer - Cloud Pipeline
Industrial Light & Magic
London
020 3751 9150
________________________________________
From: james [garftd@verizon.net]
Sent: 11 May 2016 17:08
To: user@mesos.apache.org
Subject: Re: distributed file systems
Hello Rodrick,
That EFS looks interesting, but I did not find the location for the
source-code/git download? I do not remember the (linux) kernel hooks
for that Distributed File System, or is completely on top of the systems
codes?
License details and I'm not sure if it's 100% opensource?
Beegfs [A] is partially opensource, but that does not fit what is needed
for experimentation. A robust community around open sources and tools,
such as github, should have been mentioned. Equally important
is a community keen on sharing and supporting other efforts to replicate
and use the components of these cluster centric codes. [B,C]
James
[A] http://www.beegfs.com/content/
[B] https://forums.aws.amazon.com/thread.jspa?threadID=217783
[C]
http://searchaws.techtarget.com/news/4500272286/Amazon-EFS-stuck-in-beta-lacks-native-Windows-support
On 05/11/2016 01:07 AM, Rodrick Brown wrote:
> Does EFS count? :-)
>
> https://aws.amazon.com/efs/
>
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ma...@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com <http://www.orchardplatform.com/>
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
>
> On May 10 2016, at 9:07 pm, james <ga...@verizon.net> wrote:
>
> Hello,
>
>
> Has anyone customized/compiled mesos and successfully deployed a mesos
> cluster to run on cephfs, orangefs [1], or any other distributed file
> systems?
>
> If so, some detail on your setup would be appreciated.
>
>
> [1]
> http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6
>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient
> of this communication, please delete it immediately and notify the
> sender by return email. Unauthorized reading, dissemination,
> distribution or copying of this communication is prohibited. This
> communication does not constitute an offer to sell or a solicitation of
> an indication of interest to purchase any loan, security or any other
> financial product or instrument, nor is it an offer to sell or a
> solicitation of an indication of interest to purchase any products or
> services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may
> not be accurate or complete and are subject to change without notice. As
> such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or
> completeness of the information contained herein. The intended recipient
> is advised to consult its own professional advisors, including those
> specializing in legal, tax and accounting matters. Orchard does not
> provide legal, tax or accounting advice.
Re: distributed file systems
Posted by james <ga...@verizon.net>.
Hello Rodrick,
That EFS looks interesting, but I did not find the location for the
source-code/git download? I do not remember the (linux) kernel hooks
for that Distributed File System, or is completely on top of the systems
codes?
License details and I'm not sure if it's 100% opensource?
Beegfs [A] is partially opensource, but that does not fit what is needed
for experimentation. A robust community around open sources and tools,
such as github, should have been mentioned. Equally important
is a community keen on sharing and supporting other efforts to replicate
and use the components of these cluster centric codes. [B,C]
James
[A] http://www.beegfs.com/content/
[B] https://forums.aws.amazon.com/thread.jspa?threadID=217783
[C]
http://searchaws.techtarget.com/news/4500272286/Amazon-EFS-stuck-in-beta-lacks-native-Windows-support
On 05/11/2016 01:07 AM, Rodrick Brown wrote:
> Does EFS count? :-)
>
> https://aws.amazon.com/efs/
>
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ma...@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com <http://www.orchardplatform.com/>
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
>
> On May 10 2016, at 9:07 pm, james <ga...@verizon.net> wrote:
>
> Hello,
>
>
> Has anyone customized/compiled mesos and successfully deployed a mesos
> cluster to run on cephfs, orangefs [1], or any other distributed file
> systems?
>
> If so, some detail on your setup would be appreciated.
>
>
> [1]
> http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-Linux-4.6
>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient
> of this communication, please delete it immediately and notify the
> sender by return email. Unauthorized reading, dissemination,
> distribution or copying of this communication is prohibited. This
> communication does not constitute an offer to sell or a solicitation of
> an indication of interest to purchase any loan, security or any other
> financial product or instrument, nor is it an offer to sell or a
> solicitation of an indication of interest to purchase any products or
> services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may
> not be accurate or complete and are subject to change without notice. As
> such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or
> completeness of the information contained herein. The intended recipient
> is advised to consult its own professional advisors, including those
> specializing in legal, tax and accounting matters. Orchard does not
> provide legal, tax or accounting advice.
Re: distributed file systems
Posted by Rodrick Brown <ro...@orchard-app.com>.
Does EFS count? :-)
https://aws.amazon.com/efs/
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
On May 10 2016, at 9:07 pm, james <garftd@verizon.net> wrote:
> Hello,
>
>
Has anyone customized/compiled mesos and successfully deployed a mesos
cluster to run on cephfs, orangefs [1], or any other distributed file
systems?
>
> If so, some detail on your setup would be appreciated.
>
>
[1]
[http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-
Linux-4.6](http://www.phoronix.com/scan.php?page=news_item&px=OrangeFS-Lands-
Linux-4.6)
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.