You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Tim St Clair <ts...@redhat.com> on 2014/10/28 15:46:11 UTC
0.21.0-pre & Spark latest
Folks -
We have some automated tests that run the latest Mesos against the latest Spark, and we've run across a series of issues in both fine, and course grained mode that I believe stem from a series of changes in the 0.21 cycle.
I'm not certain if anyone "owns" this integration, but we should probably ensure it's fixed before we push out 0.21.
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: 0.21.0-pre & Spark latest
Posted by Timothy Chen <tn...@gmail.com>.
Hi RJ,
I see, are you or the team on working on this problem already? If not
I'd like to take a look as well.
Tim
On Tue, Oct 28, 2014 at 8:47 AM, RJ Nowling <rn...@redhat.com> wrote:
> Hi Tim,
>
> The integration test is simply to open the spark shell (Spark 1.1.0) using mesos 0.21 in coarse-grained mode. We didn't even have to run any commands.
>
> RJ
>
> ----- Original Message -----
>> From: "Timothy Chen" <tn...@gmail.com>
>> To: dev@mesos.apache.org
>> Cc: user@mesos.apache.org, "RJ Nowling" <rn...@redhat.com>, "Erik Erlandson" <ej...@redhat.com>
>> Sent: Tuesday, October 28, 2014 11:40:19 AM
>> Subject: Re: 0.21.0-pre & Spark latest
>>
>> Hi Tim,
>>
>> Thanks for doing the integration tests, that's something that I wanted to do
>> but never got to yet.
>>
>> I have great interest ensuring spark and mesos work, and I know Brenden as
>> well does.
>>
>> I have been tracking these spark mesos problems with spark jira and labeling
>> them mesos. Can you create these bugs on jira and we can dig more on each
>> one?
>>
>> Also is this integration test automated?
>>
>> Thanks!
>>
>> Tim
>>
>> > On Oct 28, 2014, at 8:03 AM, Tim St Clair <ts...@redhat.com> wrote:
>> >
>> > inline
>> >
>> > ----- Original Message -----
>> >
>> >> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
>> >> To: user@mesos.apache.org
>> >> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling"
>> >> <rn...@redhat.com>,
>> >> "Erik Erlandson" <ej...@redhat.com>
>> >> Sent: Tuesday, October 28, 2014 9:51:58 AM
>> >> Subject: Re: 0.21.0-pre & Spark latest
>> >
>> >> Since we've recently adopted Spark, I'll second Tim's comment. We had an
>> >> issue with 0.20.1 that was possibly related to Spark[1], so it's important
>> >> for us to get this stuff fixed in 0.21.0.
>> >
>> >> Tim, can you elaborate on the issues you saw? Have you tested with my
>> >> recent
>> >> Spark patches[2][3]?
>> >
>> > We are building against Spark 1.1.0 unpatched:
>> > - Fine grained mode appears broken.
>> >
>> > - Course grained mode appears to work via normal runs, but crashes in the
>> > REPL.
>> > http://fpaste.org/145782/14506564/
>> >
>> >> [1]: https://issues.apache.org/jira/browse/MESOS-1973
>> >> [2]: https://github.com/apache/spark/pull/2401
>> >> [3]: https://github.com/apache/spark/pull/2453
>> >
>> >> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com >
>> >> wrote:
>> >
>> >>> Folks -
>> >
>> >>> We have some automated tests that run the latest Mesos against the latest
>> >>> Spark, and we've run across a series of issues in both fine, and course
>> >>> grained mode that I believe stem from a series of changes in the 0.21
>> >>> cycle.
>> >
>> >>> I'm not certain if anyone "owns" this integration, but we should probably
>> >>> ensure it's fixed before we push out 0.21.
>> >
>> >>> --
>> >>
>> >>> Cheers,
>> >>
>> >>> Timothy St. Clair
>> >>
>> >>> Red Hat Inc.
>> >
>> > --
>> > Cheers,
>> > Timothy St. Clair
>> > Red Hat Inc.
>>
Re: 0.21.0-pre & Spark latest
Posted by Timothy Chen <tn...@gmail.com>.
Hi RJ,
I see, are you or the team on working on this problem already? If not
I'd like to take a look as well.
Tim
On Tue, Oct 28, 2014 at 8:47 AM, RJ Nowling <rn...@redhat.com> wrote:
> Hi Tim,
>
> The integration test is simply to open the spark shell (Spark 1.1.0) using mesos 0.21 in coarse-grained mode. We didn't even have to run any commands.
>
> RJ
>
> ----- Original Message -----
>> From: "Timothy Chen" <tn...@gmail.com>
>> To: dev@mesos.apache.org
>> Cc: user@mesos.apache.org, "RJ Nowling" <rn...@redhat.com>, "Erik Erlandson" <ej...@redhat.com>
>> Sent: Tuesday, October 28, 2014 11:40:19 AM
>> Subject: Re: 0.21.0-pre & Spark latest
>>
>> Hi Tim,
>>
>> Thanks for doing the integration tests, that's something that I wanted to do
>> but never got to yet.
>>
>> I have great interest ensuring spark and mesos work, and I know Brenden as
>> well does.
>>
>> I have been tracking these spark mesos problems with spark jira and labeling
>> them mesos. Can you create these bugs on jira and we can dig more on each
>> one?
>>
>> Also is this integration test automated?
>>
>> Thanks!
>>
>> Tim
>>
>> > On Oct 28, 2014, at 8:03 AM, Tim St Clair <ts...@redhat.com> wrote:
>> >
>> > inline
>> >
>> > ----- Original Message -----
>> >
>> >> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
>> >> To: user@mesos.apache.org
>> >> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling"
>> >> <rn...@redhat.com>,
>> >> "Erik Erlandson" <ej...@redhat.com>
>> >> Sent: Tuesday, October 28, 2014 9:51:58 AM
>> >> Subject: Re: 0.21.0-pre & Spark latest
>> >
>> >> Since we've recently adopted Spark, I'll second Tim's comment. We had an
>> >> issue with 0.20.1 that was possibly related to Spark[1], so it's important
>> >> for us to get this stuff fixed in 0.21.0.
>> >
>> >> Tim, can you elaborate on the issues you saw? Have you tested with my
>> >> recent
>> >> Spark patches[2][3]?
>> >
>> > We are building against Spark 1.1.0 unpatched:
>> > - Fine grained mode appears broken.
>> >
>> > - Course grained mode appears to work via normal runs, but crashes in the
>> > REPL.
>> > http://fpaste.org/145782/14506564/
>> >
>> >> [1]: https://issues.apache.org/jira/browse/MESOS-1973
>> >> [2]: https://github.com/apache/spark/pull/2401
>> >> [3]: https://github.com/apache/spark/pull/2453
>> >
>> >> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com >
>> >> wrote:
>> >
>> >>> Folks -
>> >
>> >>> We have some automated tests that run the latest Mesos against the latest
>> >>> Spark, and we've run across a series of issues in both fine, and course
>> >>> grained mode that I believe stem from a series of changes in the 0.21
>> >>> cycle.
>> >
>> >>> I'm not certain if anyone "owns" this integration, but we should probably
>> >>> ensure it's fixed before we push out 0.21.
>> >
>> >>> --
>> >>
>> >>> Cheers,
>> >>
>> >>> Timothy St. Clair
>> >>
>> >>> Red Hat Inc.
>> >
>> > --
>> > Cheers,
>> > Timothy St. Clair
>> > Red Hat Inc.
>>
Re: 0.21.0-pre & Spark latest
Posted by Timothy Chen <tn...@gmail.com>.
Hi Tim,
Thanks for doing the integration tests, that's something that I wanted to do but never got to yet.
I have great interest ensuring spark and mesos work, and I know Brenden as well does.
I have been tracking these spark mesos problems with spark jira and labeling them mesos. Can you create these bugs on jira and we can dig more on each one?
Also is this integration test automated?
Thanks!
Tim
> On Oct 28, 2014, at 8:03 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> inline
>
> ----- Original Message -----
>
>> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
>> To: user@mesos.apache.org
>> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling" <rn...@redhat.com>,
>> "Erik Erlandson" <ej...@redhat.com>
>> Sent: Tuesday, October 28, 2014 9:51:58 AM
>> Subject: Re: 0.21.0-pre & Spark latest
>
>> Since we've recently adopted Spark, I'll second Tim's comment. We had an
>> issue with 0.20.1 that was possibly related to Spark[1], so it's important
>> for us to get this stuff fixed in 0.21.0.
>
>> Tim, can you elaborate on the issues you saw? Have you tested with my recent
>> Spark patches[2][3]?
>
> We are building against Spark 1.1.0 unpatched:
> - Fine grained mode appears broken.
>
> - Course grained mode appears to work via normal runs, but crashes in the REPL.
> http://fpaste.org/145782/14506564/
>
>> [1]: https://issues.apache.org/jira/browse/MESOS-1973
>> [2]: https://github.com/apache/spark/pull/2401
>> [3]: https://github.com/apache/spark/pull/2453
>
>> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com > wrote:
>
>>> Folks -
>
>>> We have some automated tests that run the latest Mesos against the latest
>>> Spark, and we've run across a series of issues in both fine, and course
>>> grained mode that I believe stem from a series of changes in the 0.21
>>> cycle.
>
>>> I'm not certain if anyone "owns" this integration, but we should probably
>>> ensure it's fixed before we push out 0.21.
>
>>> --
>>
>>> Cheers,
>>
>>> Timothy St. Clair
>>
>>> Red Hat Inc.
>
> --
> Cheers,
> Timothy St. Clair
> Red Hat Inc.
Re: 0.21.0-pre & Spark latest
Posted by Timothy Chen <tn...@gmail.com>.
Hi Tim,
Thanks for doing the integration tests, that's something that I wanted to do but never got to yet.
I have great interest ensuring spark and mesos work, and I know Brenden as well does.
I have been tracking these spark mesos problems with spark jira and labeling them mesos. Can you create these bugs on jira and we can dig more on each one?
Also is this integration test automated?
Thanks!
Tim
> On Oct 28, 2014, at 8:03 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> inline
>
> ----- Original Message -----
>
>> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
>> To: user@mesos.apache.org
>> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling" <rn...@redhat.com>,
>> "Erik Erlandson" <ej...@redhat.com>
>> Sent: Tuesday, October 28, 2014 9:51:58 AM
>> Subject: Re: 0.21.0-pre & Spark latest
>
>> Since we've recently adopted Spark, I'll second Tim's comment. We had an
>> issue with 0.20.1 that was possibly related to Spark[1], so it's important
>> for us to get this stuff fixed in 0.21.0.
>
>> Tim, can you elaborate on the issues you saw? Have you tested with my recent
>> Spark patches[2][3]?
>
> We are building against Spark 1.1.0 unpatched:
> - Fine grained mode appears broken.
>
> - Course grained mode appears to work via normal runs, but crashes in the REPL.
> http://fpaste.org/145782/14506564/
>
>> [1]: https://issues.apache.org/jira/browse/MESOS-1973
>> [2]: https://github.com/apache/spark/pull/2401
>> [3]: https://github.com/apache/spark/pull/2453
>
>> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com > wrote:
>
>>> Folks -
>
>>> We have some automated tests that run the latest Mesos against the latest
>>> Spark, and we've run across a series of issues in both fine, and course
>>> grained mode that I believe stem from a series of changes in the 0.21
>>> cycle.
>
>>> I'm not certain if anyone "owns" this integration, but we should probably
>>> ensure it's fixed before we push out 0.21.
>
>>> --
>>
>>> Cheers,
>>
>>> Timothy St. Clair
>>
>>> Red Hat Inc.
>
> --
> Cheers,
> Timothy St. Clair
> Red Hat Inc.
Re: 0.21.0-pre & Spark latest
Posted by Tim St Clair <ts...@redhat.com>.
inline
----- Original Message -----
> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
> To: user@mesos.apache.org
> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling" <rn...@redhat.com>,
> "Erik Erlandson" <ej...@redhat.com>
> Sent: Tuesday, October 28, 2014 9:51:58 AM
> Subject: Re: 0.21.0-pre & Spark latest
> Since we've recently adopted Spark, I'll second Tim's comment. We had an
> issue with 0.20.1 that was possibly related to Spark[1], so it's important
> for us to get this stuff fixed in 0.21.0.
> Tim, can you elaborate on the issues you saw? Have you tested with my recent
> Spark patches[2][3]?
We are building against Spark 1.1.0 unpatched:
- Fine grained mode appears broken.
- Course grained mode appears to work via normal runs, but crashes in the REPL.
http://fpaste.org/145782/14506564/
> [1]: https://issues.apache.org/jira/browse/MESOS-1973
> [2]: https://github.com/apache/spark/pull/2401
> [3]: https://github.com/apache/spark/pull/2453
> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com > wrote:
> > Folks -
>
> > We have some automated tests that run the latest Mesos against the latest
> > Spark, and we've run across a series of issues in both fine, and course
> > grained mode that I believe stem from a series of changes in the 0.21
> > cycle.
>
> > I'm not certain if anyone "owns" this integration, but we should probably
> > ensure it's fixed before we push out 0.21.
>
> > --
>
> > Cheers,
>
> > Timothy St. Clair
>
> > Red Hat Inc.
>
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: 0.21.0-pre & Spark latest
Posted by Tim St Clair <ts...@redhat.com>.
inline
----- Original Message -----
> From: "Brenden Matthews" <br...@airbedandbreakfast.com>
> To: user@mesos.apache.org
> Cc: "mesos-devel" <de...@mesos.apache.org>, "RJ Nowling" <rn...@redhat.com>,
> "Erik Erlandson" <ej...@redhat.com>
> Sent: Tuesday, October 28, 2014 9:51:58 AM
> Subject: Re: 0.21.0-pre & Spark latest
> Since we've recently adopted Spark, I'll second Tim's comment. We had an
> issue with 0.20.1 that was possibly related to Spark[1], so it's important
> for us to get this stuff fixed in 0.21.0.
> Tim, can you elaborate on the issues you saw? Have you tested with my recent
> Spark patches[2][3]?
We are building against Spark 1.1.0 unpatched:
- Fine grained mode appears broken.
- Course grained mode appears to work via normal runs, but crashes in the REPL.
http://fpaste.org/145782/14506564/
> [1]: https://issues.apache.org/jira/browse/MESOS-1973
> [2]: https://github.com/apache/spark/pull/2401
> [3]: https://github.com/apache/spark/pull/2453
> On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair < tstclair@redhat.com > wrote:
> > Folks -
>
> > We have some automated tests that run the latest Mesos against the latest
> > Spark, and we've run across a series of issues in both fine, and course
> > grained mode that I believe stem from a series of changes in the 0.21
> > cycle.
>
> > I'm not certain if anyone "owns" this integration, but we should probably
> > ensure it's fixed before we push out 0.21.
>
> > --
>
> > Cheers,
>
> > Timothy St. Clair
>
> > Red Hat Inc.
>
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: 0.21.0-pre & Spark latest
Posted by Brenden Matthews <br...@airbedandbreakfast.com>.
Since we've recently adopted Spark, I'll second Tim's comment. We had an
issue with 0.20.1 that was possibly related to Spark[1], so it's important
for us to get this stuff fixed in 0.21.0.
Tim, can you elaborate on the issues you saw? Have you tested with my
recent Spark patches[2][3]?
[1]: https://issues.apache.org/jira/browse/MESOS-1973
[2]: https://github.com/apache/spark/pull/2401
[3]: https://github.com/apache/spark/pull/2453
On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair <ts...@redhat.com> wrote:
> Folks -
>
> We have some automated tests that run the latest Mesos against the latest
> Spark, and we've run across a series of issues in both fine, and course
> grained mode that I believe stem from a series of changes in the 0.21 cycle.
>
> I'm not certain if anyone "owns" this integration, but we should probably
> ensure it's fixed before we push out 0.21.
>
> --
> Cheers,
> Timothy St. Clair
> Red Hat Inc.
>
Re: 0.21.0-pre & Spark latest
Posted by Brenden Matthews <br...@airbedandbreakfast.com>.
Since we've recently adopted Spark, I'll second Tim's comment. We had an
issue with 0.20.1 that was possibly related to Spark[1], so it's important
for us to get this stuff fixed in 0.21.0.
Tim, can you elaborate on the issues you saw? Have you tested with my
recent Spark patches[2][3]?
[1]: https://issues.apache.org/jira/browse/MESOS-1973
[2]: https://github.com/apache/spark/pull/2401
[3]: https://github.com/apache/spark/pull/2453
On Tue, Oct 28, 2014 at 7:46 AM, Tim St Clair <ts...@redhat.com> wrote:
> Folks -
>
> We have some automated tests that run the latest Mesos against the latest
> Spark, and we've run across a series of issues in both fine, and course
> grained mode that I believe stem from a series of changes in the 0.21 cycle.
>
> I'm not certain if anyone "owns" this integration, but we should probably
> ensure it's fixed before we push out 0.21.
>
> --
> Cheers,
> Timothy St. Clair
> Red Hat Inc.
>