You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Paul Mackles <pa...@loopr.com> on 2013/09/28 01:55:01 UTC
mesos/hadoop
Hi - Going by what is currently in the apache git repository, what is the
"recommended" combination of mesos and mesos-hadoop to use?
I originally tried the 0.13.0 tarball which appears to include some version
of mesos-hadoop. While I was able to build it and run some jobs, the task
tracker processes launched by mesos keep on running even after my job
completes. Based on the job-tracker logs, there task-trackers are not
sending the job-tracker any heartbeats so the job-tracker doesn't try to
stop them (even though it reuses them on subsequent jobs).
Based on my interpretation of
https://issues.apache.org/jira/browse/MESOS-618, it sounds like I should be
using the mesos/hadoop project on github. I managed to get that built from
trunk but I am not having nearly as much luck getting mesos built from the
trunk.
So what combination of mesos and mesos-hadoop is working for folks?
--
Thanks,
Paul
Re: mesos/hadoop
Posted by Paul Mackles <pa...@loopr.com>.
H Benjamin - Thanks for the quick response. I am keeping notes and will
share with the list after I have had a chance to clean them up.
Will try 0.14.0-rc4.
Thanks again,
Paul
On Fri, Sep 27, 2013 at 7:57 PM, Benjamin Hindman <
benjamin.hindman@gmail.com> wrote:
> Hey Paul,
>
> Feel free to share your build issues, we'd love to help.
>
> In the mean time, I suggest trying the git tag 0.14.0-rc4 with the
> mesos/hadoop project on Github.
>
> Ben.
>
>
>
>
> On Fri, Sep 27, 2013 at 4:55 PM, Paul Mackles <pa...@loopr.com> wrote:
>
>> Hi - Going by what is currently in the apache git repository, what is the
>> "recommended" combination of mesos and mesos-hadoop to use?
>>
>> I originally tried the 0.13.0 tarball which appears to include some
>> version of mesos-hadoop. While I was able to build it and run some jobs,
>> the task tracker processes launched by mesos keep on running even after my
>> job completes. Based on the job-tracker logs, there task-trackers are not
>> sending the job-tracker any heartbeats so the job-tracker doesn't try to
>> stop them (even though it reuses them on subsequent jobs).
>>
>> Based on my interpretation of
>> https://issues.apache.org/jira/browse/MESOS-618, it sounds like I should
>> be using the mesos/hadoop project on github. I managed to get that built
>> from trunk but I am not having nearly as much luck getting mesos built from
>> the trunk.
>>
>> So what combination of mesos and mesos-hadoop is working for folks?
>>
>> --
>> Thanks,
>> Paul
>>
>
>
--
Thanks,
Paul
Re: mesos/hadoop
Posted by Benjamin Hindman <be...@gmail.com>.
Hi Du,
Yes, Spark 0.8.0 should work with Mesos 0.14.0-rc4 (as well as the what's
in master).
Have you tried 'make maven-install' for installing Mesos locally in order
to build Spark? You'll need to update the project/SparkBuild.scala to point
to the local maven repository (I think).
Ben.
On Fri, Sep 27, 2013 at 5:20 PM, Du Li <du...@ericsson.com> wrote:
> Hi Ben,
>
> Similarly, what's the recommended combination of mesos/spark? Will there
> be problem using Head of mesos (0.15.0) and spark (0.8.0-SNAPSHOT)? I am
> trying to figure out how to do mvn install on mesos HEAD so that it can be
> used to build spark. Any hint would help.
>
> Thanks
> Du
>
> From: Benjamin Hindman <be...@gmail.com>
> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Date: Friday, September 27, 2013 4:57 PM
> To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Subject: Re: mesos/hadoop
>
> Hey Paul,
>
> Feel free to share your build issues, we'd love to help.
>
> In the mean time, I suggest trying the git tag 0.14.0-rc4 with the
> mesos/hadoop project on Github.
>
> Ben.
>
>
>
>
> On Fri, Sep 27, 2013 at 4:55 PM, Paul Mackles <pa...@loopr.com> wrote:
>
>> Hi - Going by what is currently in the apache git repository, what is the
>> "recommended" combination of mesos and mesos-hadoop to use?
>>
>> I originally tried the 0.13.0 tarball which appears to include some
>> version of mesos-hadoop. While I was able to build it and run some jobs,
>> the task tracker processes launched by mesos keep on running even after my
>> job completes. Based on the job-tracker logs, there task-trackers are not
>> sending the job-tracker any heartbeats so the job-tracker doesn't try to
>> stop them (even though it reuses them on subsequent jobs).
>>
>> Based on my interpretation of
>> https://issues.apache.org/jira/browse/MESOS-618, it sounds like I should
>> be using the mesos/hadoop project on github. I managed to get that built
>> from trunk but I am not having nearly as much luck getting mesos built from
>> the trunk.
>>
>> So what combination of mesos and mesos-hadoop is working for folks?
>>
>> --
>> Thanks,
>> Paul
>>
>
>
Re: mesos/hadoop
Posted by Du Li <du...@ericsson.com>.
Hi Ben,
Similarly, what's the recommended combination of mesos/spark? Will there be problem using Head of mesos (0.15.0) and spark (0.8.0-SNAPSHOT)? I am trying to figure out how to do mvn install on mesos HEAD so that it can be used to build spark. Any hint would help.
Thanks
Du
From: Benjamin Hindman <be...@gmail.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Friday, September 27, 2013 4:57 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: mesos/hadoop
Hey Paul,
Feel free to share your build issues, we'd love to help.
In the mean time, I suggest trying the git tag 0.14.0-rc4 with the mesos/hadoop project on Github.
Ben.
On Fri, Sep 27, 2013 at 4:55 PM, Paul Mackles <pa...@loopr.com>> wrote:
Hi - Going by what is currently in the apache git repository, what is the "recommended" combination of mesos and mesos-hadoop to use?
I originally tried the 0.13.0 tarball which appears to include some version of mesos-hadoop. While I was able to build it and run some jobs, the task tracker processes launched by mesos keep on running even after my job completes. Based on the job-tracker logs, there task-trackers are not sending the job-tracker any heartbeats so the job-tracker doesn't try to stop them (even though it reuses them on subsequent jobs).
Based on my interpretation of https://issues.apache.org/jira/browse/MESOS-618, it sounds like I should be using the mesos/hadoop project on github. I managed to get that built from trunk but I am not having nearly as much luck getting mesos built from the trunk.
So what combination of mesos and mesos-hadoop is working for folks?
--
Thanks,
Paul
Re: mesos/hadoop
Posted by Benjamin Hindman <be...@gmail.com>.
Hey Paul,
Feel free to share your build issues, we'd love to help.
In the mean time, I suggest trying the git tag 0.14.0-rc4 with the
mesos/hadoop project on Github.
Ben.
On Fri, Sep 27, 2013 at 4:55 PM, Paul Mackles <pa...@loopr.com> wrote:
> Hi - Going by what is currently in the apache git repository, what is the
> "recommended" combination of mesos and mesos-hadoop to use?
>
> I originally tried the 0.13.0 tarball which appears to include some
> version of mesos-hadoop. While I was able to build it and run some jobs,
> the task tracker processes launched by mesos keep on running even after my
> job completes. Based on the job-tracker logs, there task-trackers are not
> sending the job-tracker any heartbeats so the job-tracker doesn't try to
> stop them (even though it reuses them on subsequent jobs).
>
> Based on my interpretation of
> https://issues.apache.org/jira/browse/MESOS-618, it sounds like I should
> be using the mesos/hadoop project on github. I managed to get that built
> from trunk but I am not having nearly as much luck getting mesos built from
> the trunk.
>
> So what combination of mesos and mesos-hadoop is working for folks?
>
> --
> Thanks,
> Paul
>