You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jianshi Huang <ji...@gmail.com> on 2015/02/11 05:43:27 UTC
Which version to use for shuffle service if I'm going to run multiple
versions of Spark
Hi,
I need to use branch-1.2 and sometimes master builds of Spark for my
project. However the officially supported Spark version by our Hadoop admin
is only 1.2.0.
So, my question is which version/build of spark-yarn-shuffle.jar should I
use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
Thanks,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Which version to use for shuffle service if I'm going to run
multiple versions of Spark
Posted by Jianshi Huang <ji...@gmail.com>.
Get it. Thanks Reynold and Andrew!
Jianshi
On Thu, Feb 12, 2015 at 12:25 AM, Andrew Or <an...@databricks.com> wrote:
> Hi Jianshi,
>
> For YARN, there may be an issue with how a recently patch changes the
> accessibility of the shuffle files by the external shuffle service:
> https://issues.apache.org/jira/browse/SPARK-5655. It is likely that you
> will hit this with 1.2.1, actually. For this reason I would have to
> recommend that you use 1.2.2 when it is released, but for now you should
> use 1.2.0 for this specific use case.
>
> -Andrew
>
> 2015-02-10 23:38 GMT-08:00 Reynold Xin <rx...@databricks.com>:
>
> I think we made the binary protocol compatible across all versions, so you
>> should be fine with using any one of them. 1.2.1 is probably the best since
>> it is the most recent stable release.
>>
>> On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I need to use branch-1.2 and sometimes master builds of Spark for my
>>> project. However the officially supported Spark version by our Hadoop admin
>>> is only 1.2.0.
>>>
>>> So, my question is which version/build of spark-yarn-shuffle.jar should
>>> I use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>>>
>>> Thanks,
>>> --
>>> Jianshi Huang
>>>
>>> LinkedIn: jianshi
>>> Twitter: @jshuang
>>> Github & Blog: http://huangjs.github.com/
>>>
>>
>>
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Which version to use for shuffle service if I'm going to run
multiple versions of Spark
Posted by Andrew Or <an...@databricks.com>.
Hi Jianshi,
For YARN, there may be an issue with how a recently patch changes the
accessibility of the shuffle files by the external shuffle service:
https://issues.apache.org/jira/browse/SPARK-5655. It is likely that you
will hit this with 1.2.1, actually. For this reason I would have to
recommend that you use 1.2.2 when it is released, but for now you should
use 1.2.0 for this specific use case.
-Andrew
2015-02-10 23:38 GMT-08:00 Reynold Xin <rx...@databricks.com>:
> I think we made the binary protocol compatible across all versions, so you
> should be fine with using any one of them. 1.2.1 is probably the best since
> it is the most recent stable release.
>
> On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I need to use branch-1.2 and sometimes master builds of Spark for my
>> project. However the officially supported Spark version by our Hadoop admin
>> is only 1.2.0.
>>
>> So, my question is which version/build of spark-yarn-shuffle.jar should I
>> use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>>
>> Thanks,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
Re: Which version to use for shuffle service if I'm going to run
multiple versions of Spark
Posted by Reynold Xin <rx...@databricks.com>.
I think we made the binary protocol compatible across all versions, so you
should be fine with using any one of them. 1.2.1 is probably the best since
it is the most recent stable release.
On Tue, Feb 10, 2015 at 8:43 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> Hi,
>
> I need to use branch-1.2 and sometimes master builds of Spark for my
> project. However the officially supported Spark version by our Hadoop admin
> is only 1.2.0.
>
> So, my question is which version/build of spark-yarn-shuffle.jar should I
> use that works for all four versions? (1.2.0, 1.2.1, 1.2,2, 1.3.0)
>
> Thanks,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>