You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jianshi Huang <ji...@gmail.com> on 2015/02/11 05:43:27 UTC

Which version to use for shuffle service if I'm going to run multiple versions of Spark

Hi,

I need to use branch-1.2 and sometimes master builds of Spark for my
project. However the officially supported Spark version by our Hadoop admin
is only 1.2.0.

So, my question is which version/build of spark-yarn-shuffle.jar should I
use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)

Thanks,
-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Re: Which version to use for shuffle service if I'm going to run multiple versions of Spark

Posted by Jianshi Huang <ji...@gmail.com>.
Get it. Thanks Reynold and Andrew!

Jianshi

On Thu, Feb 12, 2015 at 12:25 AM, Andrew Or <an...@databricks.com> wrote:

> Hi Jianshi,
>
> For YARN, there may be an issue with how a recently patch changes the
> accessibility of the shuffle files by the external shuffle service:
> https://issues.apache.org/jira/browse/SPARK-5655. It is likely that you
> will hit this with 1.2.1, actually. For this reason I would have to
> recommend that you use 1.2.2 when it is released, but for now you should
> use 1.2.0 for this specific use case.
>
> -Andrew
>
> 2015-02-10 23:38 GMT-08:00 Reynold Xin <rx...@databricks.com>:
>
> I think we made the binary protocol compatible across all versions, so you
>> should be fine with using any one of them. 1.2.1 is probably the best since
>> it is the most recent stable release.
>>
>> On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I need to use branch-1.2 and sometimes master builds of Spark for my
>>> project. However the officially supported Spark version by our Hadoop admin
>>> is only 1.2.0.
>>>
>>> So, my question is which version/build of spark-yarn-shuffle.jar should
>>> I use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>>>
>>> Thanks,
>>> --
>>> Jianshi Huang
>>>
>>> LinkedIn: jianshi
>>> Twitter: @jshuang
>>> Github & Blog: http://huangjs.github.com/
>>>
>>
>>
>


-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Re: Which version to use for shuffle service if I'm going to run multiple versions of Spark

Posted by Andrew Or <an...@databricks.com>.
Hi Jianshi,

For YARN, there may be an issue with how a recently patch changes the
accessibility of the shuffle files by the external shuffle service:
https://issues.apache.org/jira/browse/SPARK-5655. It is likely that you
will hit this with 1.2.1, actually. For this reason I would have to
recommend that you use 1.2.2 when it is released, but for now you should
use 1.2.0 for this specific use case.

-Andrew

2015-02-10 23:38 GMT-08:00 Reynold Xin <rx...@databricks.com>:

> I think we made the binary protocol compatible across all versions, so you
> should be fine with using any one of them. 1.2.1 is probably the best since
> it is the most recent stable release.
>
> On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I need to use branch-1.2 and sometimes master builds of Spark for my
>> project. However the officially supported Spark version by our Hadoop admin
>> is only 1.2.0.
>>
>> So, my question is which version/build of spark-yarn-shuffle.jar should I
>> use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>>
>> Thanks,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>

Re: Which version to use for shuffle service if I'm going to run multiple versions of Spark

Posted by Reynold Xin <rx...@databricks.com>.
I think we made the binary protocol compatible across all versions, so you
should be fine with using any one of them. 1.2.1 is probably the best since
it is the most recent stable release.

On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
wrote:

> Hi,
>
> I need to use branch-1.2 and sometimes master builds of Spark for my
> project. However the officially supported Spark version by our Hadoop admin
> is only 1.2.0.
>
> So, my question is which version/build of spark-yarn-shuffle.jar should I
> use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>
> Thanks,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>