Cross-compatibility of YARN shuffle service

[Matt Cheah <mc...@palantir.com>]

Hi everyone,

I am considering moving from Spark-Standalone to YARN. The context is that
there are multiple Spark applications that are using different versions of
Spark that all want to use the same YARN cluster.

My question is: if I use a single Spark YARN shuffle service jar on the Node
Manager, will the service work properly with all of the Spark applications,
regardless of the specific versions of the applications? Or, is it it the
case that, if I want to use the external shuffle service, I need to have all
of my applications using the same version of Spark?

Thanks,

-Matt Cheah

Re: Cross-compatibility of YARN shuffle service

[Sandy Ryza <sa...@cloudera.com>]

Hi Matt,

I'm not sure whether we have documented compatibility guidelines here.
However, a strong goal is to keep the external shuffle service compatible
so that many versions of Spark can run against the same shuffle service.

-Sandy

On Wed, Mar 25, 2015 at 6:44 PM, Matt Cheah <mc...@palantir.com> wrote:

> Hi everyone,
>
> I am considering moving from Spark-Standalone to YARN. The context is that
> there are multiple Spark applications that are using different versions of
> Spark that all want to use the same YARN cluster.
>
> My question is: if I use a single Spark YARN shuffle service jar on the
> Node Manager, will the service work properly with all of the Spark
> applications, regardless of the specific versions of the applications? Or,
> is it it the case that, if I want to use the external shuffle service, I
> need to have all of my applications using the same version of Spark?
>
> Thanks,
>
> -Matt Cheah
>