You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Benjamin Mahler <be...@gmail.com> on 2014/01/10 20:01:10 UTC
Distcc on Mesos.
I wrote a simple a simple framework for doing distributed C++ compilation
through distcc:
https://github.com/mesos/mesos-distcc
Distcc is a distributed C++ compiler hosted here:
https://code.google.com/p/distcc/
The mesos-distcc tool will dynamically create a distcc cluster for the
duration of your make invocation, making it possible to use your Mesos
resources to speed up your C++ builds.
Interestingly, I was able to build the mesos project itself using this tool
with around a 2x speedup over a single 4 core machine (see the README on
github).
Note that distcc's 'pump' mode does not work for mesos due to the following
distcc issues, which restricts the amount of speedup possible:
https://code.google.com/p/distcc/issues/detail?id=40
https://code.google.com/p/distcc/issues/detail?id=16
For projects that can use 'pump' mode and that have a wide build
parallelism, this should prove additionally beneficial. See here for
benchmarks:
http://distcc.googlecode.com/svn/trunk/doc/web/benchmark.html
I hope this serves as a helpful example of a simple framework. For those
spending their time waiting on c++ builds, I hope this proves to be a
useful tool!
Ben
Re: Distcc on Mesos.
Posted by Tobias Knaup <to...@knaup.me>.
This is great! We were thinking about building the same thing :)
On Fri, Jan 10, 2014 at 11:01 AM, Benjamin Mahler <benjamin.mahler@gmail.com
> wrote:
> I wrote a simple a simple framework for doing distributed C++ compilation
> through distcc:
> https://github.com/mesos/mesos-distcc
>
> Distcc is a distributed C++ compiler hosted here:
> https://code.google.com/p/distcc/
>
> The mesos-distcc tool will dynamically create a distcc cluster for the
> duration of your make invocation, making it possible to use your Mesos
> resources to speed up your C++ builds.
>
> Interestingly, I was able to build the mesos project itself using this tool
> with around a 2x speedup over a single 4 core machine (see the README on
> github).
>
> Note that distcc's 'pump' mode does not work for mesos due to the following
> distcc issues, which restricts the amount of speedup possible:
> https://code.google.com/p/distcc/issues/detail?id=40
> https://code.google.com/p/distcc/issues/detail?id=16
>
> For projects that can use 'pump' mode and that have a wide build
> parallelism, this should prove additionally beneficial. See here for
> benchmarks:
> http://distcc.googlecode.com/svn/trunk/doc/web/benchmark.html
>
> I hope this serves as a helpful example of a simple framework. For those
> spending their time waiting on c++ builds, I hope this proves to be a
> useful tool!
>
> Ben
>
Re: Distcc on Mesos.
Posted by Tobias Knaup <to...@knaup.me>.
This is great! We were thinking about building the same thing :)
On Fri, Jan 10, 2014 at 11:01 AM, Benjamin Mahler <benjamin.mahler@gmail.com
> wrote:
> I wrote a simple a simple framework for doing distributed C++ compilation
> through distcc:
> https://github.com/mesos/mesos-distcc
>
> Distcc is a distributed C++ compiler hosted here:
> https://code.google.com/p/distcc/
>
> The mesos-distcc tool will dynamically create a distcc cluster for the
> duration of your make invocation, making it possible to use your Mesos
> resources to speed up your C++ builds.
>
> Interestingly, I was able to build the mesos project itself using this tool
> with around a 2x speedup over a single 4 core machine (see the README on
> github).
>
> Note that distcc's 'pump' mode does not work for mesos due to the following
> distcc issues, which restricts the amount of speedup possible:
> https://code.google.com/p/distcc/issues/detail?id=40
> https://code.google.com/p/distcc/issues/detail?id=16
>
> For projects that can use 'pump' mode and that have a wide build
> parallelism, this should prove additionally beneficial. See here for
> benchmarks:
> http://distcc.googlecode.com/svn/trunk/doc/web/benchmark.html
>
> I hope this serves as a helpful example of a simple framework. For those
> spending their time waiting on c++ builds, I hope this proves to be a
> useful tool!
>
> Ben
>