You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andreas Fritzler <an...@gmail.com> on 2015/10/05 15:22:42 UTC
[Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Hi,
I was just wondering, if it is possible to register multiple versions of
the aux-services with YARN as described in the documentation:
1. In the yarn-site.xml on each node, add spark_shuffle to
yarn.nodemanager.aux-services, then set
yarn.nodemanager.aux-services.spark_shuffle.class to
org.apache.spark.network.yarn.YarnShuffleService. Additionally, set all
relevantspark.shuffle.service.* configurations
<http://spark.apache.org/docs/latest/configuration.html>.
The reason for the question is: I am trying to run multiple versions of
Spark in parallel. Does anybody have any experience on how such a dual
version operation holds up in terms of downward-compatibility?
Maybe sticking to the latest version of the aux-service will do the trick?
Regards,
Andreas
[1]
http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Andreas Fritzler <an...@gmail.com>.
Hi Andrew,
thanks a lot for the clarification!
Regards,
Andreas
On Tue, Oct 6, 2015 at 2:23 AM, Andrew Or <an...@databricks.com> wrote:
> Hi all,
>
> Both the history server and the shuffle service are backward compatible,
> but not forward compatible. This means as long as you have the latest
> version of history server / shuffle service running in your cluster then
> you're fine (you don't need multiple of them).
>
> That said, an old shuffle service (e.g. 1.2) also happens to work with say
> Spark 1.4 because the shuffle file formats haven't changed. However, there
> are no guarantees that this will remain the case.
>
> -Andrew
>
> 2015-10-05 16:37 GMT-07:00 Alex Rovner <al...@magnetic.com>:
>
>> We are running CDH 5.4 with Spark 1.3 as our main version and that
>> version is configured to use the external shuffling service. We have also
>> installed Spark 1.5 and have configured it not to use the external
>> shuffling service and that works well for us so far. I would be interested
>> myself how to configure multiple versions to use the same shuffling service.
>>
>> *Alex Rovner*
>> *Director, Data Engineering *
>> *o:* 646.759.0052
>>
>> * <http://www.magnetic.com/>*
>>
>> On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler <
>> andreas.fritzler@gmail.com> wrote:
>>
>>> Hi Steve, Alex,
>>>
>>> how do you handle the distribution and configuration of
>>> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2
>>> different Spark versions?
>>>
>>> Regards,
>>> Andreas
>>>
>>> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>>
>>>> > On 5 Oct 2015, at 16:48, Alex Rovner <al...@magnetic.com>
>>>> wrote:
>>>> >
>>>> > Hey Steve,
>>>> >
>>>> > Are you referring to the 1.5 version of the history server?
>>>> >
>>>>
>>>>
>>>> Yes. I should warn, however, that there's no guarantee that a history
>>>> server running the 1.4 code will handle the histories of a 1.5+ job. In
>>>> fact, I'm fairly confident it won't, as the events to get replayed are
>>>> different.
>>>>
>>>
>>>
>>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Deenar Toraskar <de...@gmail.com>.
Hi guys
1. >> Add this jar to the classpath of all NodeManagers in your cluster.
A related question on configuration of the auxillary shuffle service. *How
do i find the classpath for NodeManager?* I tried finding all places where
the existing mapreduce shuffle jars are present and place the spark yarn
shuffle jar in the same location, but with no success.
$ find . -name *shuffle*.jar
./hadoop/client/hadoop-mapreduce-client-shuffle.jar
./hadoop/client/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar
./hadoop/client/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar
./hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar
./hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar
./falcon/client/lib/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar
./oozie/libserver/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar
./oozie/libtools/hadoop-mapreduce-client-shuffle-2.7.1.2.3.2.0-2950.jar
./spark/lib/spark-1.4.1.2.3.2.0-2950-yarn-shuffle.jar
Regards
Deenar
On 7 October 2015 at 01:27, Alex Rovner <al...@magnetic.com> wrote:
> Thank you all for your help.
>
> *Alex Rovner*
> *Director, Data Engineering *
> *o:* 646.759.0052
>
> * <http://www.magnetic.com/>*
>
> On Tue, Oct 6, 2015 at 11:17 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> On 6 Oct 2015, at 01:23, Andrew Or <an...@databricks.com> wrote:
>>
>> Both the history server and the shuffle service are backward compatible,
>> but not forward compatible. This means as long as you have the latest
>> version of history server / shuffle service running in your cluster then
>> you're fine (you don't need multiple of them).
>>
>>
>> FWIW I've just created a JIRA on tracking/reporting version mismatch on
>> history server playback better:
>> https://issues.apache.org/jira/browse/SPARK-10950
>>
>> Even though the UI can't be expected to playback later histories, it
>> could be possible to report the issue in a way that users can act on "run a
>> later version", rather than raise support calls.
>>
>>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Alex Rovner <al...@magnetic.com>.
Thank you all for your help.
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052
* <http://www.magnetic.com/>*
On Tue, Oct 6, 2015 at 11:17 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> On 6 Oct 2015, at 01:23, Andrew Or <an...@databricks.com> wrote:
>
> Both the history server and the shuffle service are backward compatible,
> but not forward compatible. This means as long as you have the latest
> version of history server / shuffle service running in your cluster then
> you're fine (you don't need multiple of them).
>
>
> FWIW I've just created a JIRA on tracking/reporting version mismatch on
> history server playback better:
> https://issues.apache.org/jira/browse/SPARK-10950
>
> Even though the UI can't be expected to playback later histories, it could
> be possible to report the issue in a way that users can act on "run a later
> version", rather than raise support calls.
>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Steve Loughran <st...@hortonworks.com>.
On 6 Oct 2015, at 01:23, Andrew Or <an...@databricks.com>> wrote:
Both the history server and the shuffle service are backward compatible, but not forward compatible. This means as long as you have the latest version of history server / shuffle service running in your cluster then you're fine (you don't need multiple of them).
FWIW I've just created a JIRA on tracking/reporting version mismatch on history server playback better: https://issues.apache.org/jira/browse/SPARK-10950
Even though the UI can't be expected to playback later histories, it could be possible to report the issue in a way that users can act on "run a later version", rather than raise support calls.
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Andrew Or <an...@databricks.com>.
Hi all,
Both the history server and the shuffle service are backward compatible,
but not forward compatible. This means as long as you have the latest
version of history server / shuffle service running in your cluster then
you're fine (you don't need multiple of them).
That said, an old shuffle service (e.g. 1.2) also happens to work with say
Spark 1.4 because the shuffle file formats haven't changed. However, there
are no guarantees that this will remain the case.
-Andrew
2015-10-05 16:37 GMT-07:00 Alex Rovner <al...@magnetic.com>:
> We are running CDH 5.4 with Spark 1.3 as our main version and that version
> is configured to use the external shuffling service. We have also installed
> Spark 1.5 and have configured it not to use the external shuffling service
> and that works well for us so far. I would be interested myself how to
> configure multiple versions to use the same shuffling service.
>
> *Alex Rovner*
> *Director, Data Engineering *
> *o:* 646.759.0052
>
> * <http://www.magnetic.com/>*
>
> On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler <
> andreas.fritzler@gmail.com> wrote:
>
>> Hi Steve, Alex,
>>
>> how do you handle the distribution and configuration of
>> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2
>> different Spark versions?
>>
>> Regards,
>> Andreas
>>
>> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>
>>> > On 5 Oct 2015, at 16:48, Alex Rovner <al...@magnetic.com> wrote:
>>> >
>>> > Hey Steve,
>>> >
>>> > Are you referring to the 1.5 version of the history server?
>>> >
>>>
>>>
>>> Yes. I should warn, however, that there's no guarantee that a history
>>> server running the 1.4 code will handle the histories of a 1.5+ job. In
>>> fact, I'm fairly confident it won't, as the events to get replayed are
>>> different.
>>>
>>
>>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Alex Rovner <al...@magnetic.com>.
We are running CDH 5.4 with Spark 1.3 as our main version and that version
is configured to use the external shuffling service. We have also installed
Spark 1.5 and have configured it not to use the external shuffling service
and that works well for us so far. I would be interested myself how to
configure multiple versions to use the same shuffling service.
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052
* <http://www.magnetic.com/>*
On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler <
andreas.fritzler@gmail.com> wrote:
> Hi Steve, Alex,
>
> how do you handle the distribution and configuration of
> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2
> different Spark versions?
>
> Regards,
> Andreas
>
> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> > On 5 Oct 2015, at 16:48, Alex Rovner <al...@magnetic.com> wrote:
>> >
>> > Hey Steve,
>> >
>> > Are you referring to the 1.5 version of the history server?
>> >
>>
>>
>> Yes. I should warn, however, that there's no guarantee that a history
>> server running the 1.4 code will handle the histories of a 1.5+ job. In
>> fact, I'm fairly confident it won't, as the events to get replayed are
>> different.
>>
>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Andreas Fritzler <an...@gmail.com>.
Hi Steve, Alex,
how do you handle the distribution and configuration of
the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2
different Spark versions?
Regards,
Andreas
On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 5 Oct 2015, at 16:48, Alex Rovner <al...@magnetic.com> wrote:
> >
> > Hey Steve,
> >
> > Are you referring to the 1.5 version of the history server?
> >
>
>
> Yes. I should warn, however, that there's no guarantee that a history
> server running the 1.4 code will handle the histories of a 1.5+ job. In
> fact, I'm fairly confident it won't, as the events to get replayed are
> different.
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Steve Loughran <st...@hortonworks.com>.
> On 5 Oct 2015, at 16:48, Alex Rovner <al...@magnetic.com> wrote:
>
> Hey Steve,
>
> Are you referring to the 1.5 version of the history server?
>
Yes. I should warn, however, that there's no guarantee that a history server running the 1.4 code will handle the histories of a 1.5+ job. In fact, I'm fairly confident it won't, as the events to get replayed are different.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Alex Rovner <al...@magnetic.com>.
Hey Steve,
Are you referring to the 1.5 version of the history server?
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052
* <http://www.magnetic.com/>*
On Mon, Oct 5, 2015 at 10:18 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 5 Oct 2015, at 15:59, Alex Rovner <al...@magnetic.com> wrote:
> >
> > I have the same question about the history server. We are trying to run
> multiple versions of Spark and are wondering if the history server is
> backwards compatible.
>
> yes, it supports the pre-1.4 "Single attempt" logs as well as the 1.4+
> multiple attempt model.
>
>
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Steve Loughran <st...@hortonworks.com>.
> On 5 Oct 2015, at 15:59, Alex Rovner <al...@magnetic.com> wrote:
>
> I have the same question about the history server. We are trying to run multiple versions of Spark and are wondering if the history server is backwards compatible.
yes, it supports the pre-1.4 "Single attempt" logs as well as the 1.4+ multiple attempt model.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Posted by Alex Rovner <al...@magnetic.com>.
I have the same question about the history server. We are trying to run
multiple versions of Spark and are wondering if the history server is
backwards compatible.
*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052
* <http://www.magnetic.com/>*
On Mon, Oct 5, 2015 at 9:22 AM, Andreas Fritzler <andreas.fritzler@gmail.com
> wrote:
> Hi,
>
> I was just wondering, if it is possible to register multiple versions of
> the aux-services with YARN as described in the documentation:
>
>
>
> 1. In the yarn-site.xml on each node, add spark_shuffle to
> yarn.nodemanager.aux-services, then set
> yarn.nodemanager.aux-services.spark_shuffle.class to
> org.apache.spark.network.yarn.YarnShuffleService. Additionally, set
> all relevantspark.shuffle.service.* configurations
> <http://spark.apache.org/docs/latest/configuration.html>.
>
> The reason for the question is: I am trying to run multiple versions of
> Spark in parallel. Does anybody have any experience on how such a dual
> version operation holds up in terms of downward-compatibility?
>
> Maybe sticking to the latest version of the aux-service will do the trick?
>
> Regards,
> Andreas
>
> [1]
> http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
>
>