You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mark Grover <ma...@apache.org> on 2016/04/18 22:51:01 UTC
YARN Shuffle service and its compatibility
Hi all,
If you don't use Spark on YARN, you probably don't need to read further.
Here's the *user scenario*:
There are going to be folks who may be interested in running two versions
of Spark (say Spark 1.6.x and Spark 2.x) on the same YARN cluster.
And, here's the *problem*:
That's all fine, should work well. However, there's one problem that
relates to the YARN shuffle service
<https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java>.
This service is run by the YARN Node Managers on all nodes of the cluster
that have YARN NMs as an auxillary service
<https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html>
.
The key question here is -
Option A: Should the user be running 2 shuffle services - one for Spark
1.6.x and one for Spark 2.x?
OR
Option B: Should the user be running only 1 shuffle service that services
both the Spark 1.6.x and Spark 2.x installs? This will likely have to be
the Spark 1.6.x shuffle service (while ensuring it's forward compatible
with Spark 2.x).
*Discussion of above options:*
A few things to note about the shuffle service:
1. Looking at the commit history, there aren't a whole of lot of changes
that go into the shuffle service, rarely ones that are incompatible.
There's only one incompatible change
<https://issues.apache.org/jira/browse/SPARK-12130> that's been made to the
shuffle service, as far as I can tell, and that too, seems fairly cosmetic.
2. Shuffle services for 1.6.x and 2.x serve very similar purpose (to
provide shuffle blocks) and can easily be just one service that does it,
even on a YARN cluster that runs both Spark 1.x and Spark 2.x.
3. The shuffle service is not version-spaced. This means that, the way the
code is currently, if we were to drop the jars for Spark1 and Spark2's
shuffle service in YARN NM's classpath, YARN NM won't be able to start both
services. It would arbitrarily pick one service to start (based on what
appears on the classpath first). Also, the service name is hardcoded
<https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java#L96>
in Spark code and that name is also not version-spaced.
Option A is arguably cleaner but it's more operational overhead and some
code relocation/shading/version-spacing/name-spacing to make it work (due
to #3 above), potentially to not a whole lot of value (given #2 above).
Option B is simpler, lean and more operationally efficient. However, that
requires that we as a community, keep Spark 1's shuffle service forward
compatible with Spark 2 i.e. don't break compatibility between Spark1's and
Spark2's shuffle service. We could even add a test (mima?) to assert that
during the life time of Spark2. If we do go down that way, we should revert
SPARK-12130 <https://issues.apache.org/jira/browse/SPARK-12130> - the only
backwards incompatible change made to Spark2 shuffle service so far.
My personal vote goes towards Option B and I think reverting SPARK-12130 is
ok. What do others think?
Thanks!
Mark
Re: YARN Shuffle service and its compatibility
Posted by Marcin Tustin <mt...@handybook.com>.
I'm good with option B at least until it blocks something utterly wonderful
(like shuffles are 10x faster).
On Mon, Apr 18, 2016 at 4:51 PM, Mark Grover <ma...@apache.org> wrote:
> Hi all,
> If you don't use Spark on YARN, you probably don't need to read further.
>
> Here's the *user scenario*:
> There are going to be folks who may be interested in running two versions
> of Spark (say Spark 1.6.x and Spark 2.x) on the same YARN cluster.
>
> And, here's the *problem*:
> That's all fine, should work well. However, there's one problem that
> relates to the YARN shuffle service
> <https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java>.
> This service is run by the YARN Node Managers on all nodes of the cluster
> that have YARN NMs as an auxillary service
> <https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html>
> .
>
> The key question here is -
> Option A: Should the user be running 2 shuffle services - one for Spark
> 1.6.x and one for Spark 2.x?
> OR
> Option B: Should the user be running only 1 shuffle service that services
> both the Spark 1.6.x and Spark 2.x installs? This will likely have to be
> the Spark 1.6.x shuffle service (while ensuring it's forward compatible
> with Spark 2.x).
>
> *Discussion of above options:*
> A few things to note about the shuffle service:
> 1. Looking at the commit history, there aren't a whole of lot of changes
> that go into the shuffle service, rarely ones that are incompatible.
> There's only one incompatible change
> <https://issues.apache.org/jira/browse/SPARK-12130> that's been made to
> the shuffle service, as far as I can tell, and that too, seems fairly
> cosmetic.
> 2. Shuffle services for 1.6.x and 2.x serve very similar purpose (to
> provide shuffle blocks) and can easily be just one service that does it,
> even on a YARN cluster that runs both Spark 1.x and Spark 2.x.
> 3. The shuffle service is not version-spaced. This means that, the way the
> code is currently, if we were to drop the jars for Spark1 and Spark2's
> shuffle service in YARN NM's classpath, YARN NM won't be able to start both
> services. It would arbitrarily pick one service to start (based on what
> appears on the classpath first). Also, the service name is hardcoded
> <https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java#L96>
> in Spark code and that name is also not version-spaced.
>
> Option A is arguably cleaner but it's more operational overhead and some
> code relocation/shading/version-spacing/name-spacing to make it work (due
> to #3 above), potentially to not a whole lot of value (given #2 above).
>
> Option B is simpler, lean and more operationally efficient. However, that
> requires that we as a community, keep Spark 1's shuffle service forward
> compatible with Spark 2 i.e. don't break compatibility between Spark1's and
> Spark2's shuffle service. We could even add a test (mima?) to assert that
> during the life time of Spark2. If we do go down that way, we should revert
> SPARK-12130 <https://issues.apache.org/jira/browse/SPARK-12130> - the
> only backwards incompatible change made to Spark2 shuffle service so far.
>
> My personal vote goes towards Option B and I think reverting SPARK-12130
> is ok. What do others think?
>
> Thanks!
> Mark
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: YARN Shuffle service and its compatibility
Posted by Mark Grover <ma...@apache.org>.
On Tue, Apr 19, 2016 at 2:26 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 18 Apr 2016, at 23:05, Marcelo Vanzin <va...@cloudera.com> wrote:
> >
> > On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <rx...@databricks.com>
> wrote:
> >> The bigger problem is that it is much easier to maintain backward
> >> compatibility rather than dictating forward compatibility. For example,
> as
> >> Marcin said, if we come up with a slightly different shuffle layout to
> >> improve shuffle performance, we wouldn't be able to do that if we want
> to
> >> allow Spark 1.6 shuffle service to read something generated by Spark
> 2.1.
> >
> > And I think that's really what Mark is proposing. Basically, "don't
> > intentionally break backwards compatibility unless it's really
> > required" (e.g. SPARK-12130). That would allow option B to work.
> >
> > If a new shuffle manager is created, then neither option A nor option
> > B would really work. Moving all the shuffle-related classes to a
> > different package, to support option A, would be really messy. At that
> > point, you're better off maintaining the new shuffle service outside
> > of YARN, which is rather messy too.
> >
>
>
> There's a WiP in YARN to move Aux NM services into their own CP, though
> that doesn't address shared native libs, such as the leveldb support that
> went into 1.6
>
>
> There's already been some fun with Jackson versions and that of Hadoop —
> SPARK-12807; something that per-service classpaths would fix.
>
> would having separate CPs allow multiple spark shuffle JARs to be loaded,
> as long as everything bonded to the right one?
I just checked out https://issues.apache.org/jira/browse/YARN-1593. It's
hard to say if it'd help or not, I wasn't able to find any design doc or
patch attached to that JIRA. If there were a way to specify different JAR
names/locations for starting the separate process, it would work but if the
start happened by pointing to a full class name, that comes back to Option
A, and we'd have to do a good chunk of name/version spacing in order to
isolate.
Re: YARN Shuffle service and its compatibility
Posted by Steve Loughran <st...@hortonworks.com>.
> On 18 Apr 2016, at 23:05, Marcelo Vanzin <va...@cloudera.com> wrote:
>
> On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <rx...@databricks.com> wrote:
>> The bigger problem is that it is much easier to maintain backward
>> compatibility rather than dictating forward compatibility. For example, as
>> Marcin said, if we come up with a slightly different shuffle layout to
>> improve shuffle performance, we wouldn't be able to do that if we want to
>> allow Spark 1.6 shuffle service to read something generated by Spark 2.1.
>
> And I think that's really what Mark is proposing. Basically, "don't
> intentionally break backwards compatibility unless it's really
> required" (e.g. SPARK-12130). That would allow option B to work.
>
> If a new shuffle manager is created, then neither option A nor option
> B would really work. Moving all the shuffle-related classes to a
> different package, to support option A, would be really messy. At that
> point, you're better off maintaining the new shuffle service outside
> of YARN, which is rather messy too.
>
There's a WiP in YARN to move Aux NM services into their own CP, though that doesn't address shared native libs, such as the leveldb support that went into 1.6
There's already been some fun with Jackson versions and that of Hadoop — SPARK-12807; something that per-service classpaths would fix.
would having separate CPs allow multiple spark shuffle JARs to be loaded, as long as everything bonded to the right one?
Re: YARN Shuffle service and its compatibility
Posted by Mark Grover <ma...@apache.org>.
Great, thanks for confirming, Reynold. Appreciate it!
On Tue, Apr 19, 2016 at 4:20 PM, Reynold Xin <rx...@databricks.com> wrote:
> I talked to Lianhui offline and he said it is not that big of a deal to
> revert the patch.
>
>
> On Tue, Apr 19, 2016 at 9:52 AM, Mark Grover <ma...@apache.org> wrote:
>
>> Thanks.
>>
>> I'm more than happy to wait for more people to chime in here but I do
>> feel that most of us are leaning towards Option B anyways. So, I created a
>> JIRA (SPARK-14731) for reverting SPARK-12130 in Spark 2.0 and file a PR
>> shortly.
>> Mark
>>
>> On Tue, Apr 19, 2016 at 7:44 AM, Tom Graves <tgraves_cs@yahoo.com.invalid
>> > wrote:
>>
>>> It would be nice if we could keep this compatible between 1.6 and 2.0 so
>>> I'm more for Option B at this point since the change made seems minor
>>> and we can change to have shuffle service do internally like Marcelo
>>> mention. Then lets try to keep compatible, but if there is a forcing
>>> function lets figure out a good way to run 2 at once.
>>>
>>>
>>> Tom
>>>
>>>
>>> On Monday, April 18, 2016 5:23 PM, Marcelo Vanzin <va...@cloudera.com>
>>> wrote:
>>>
>>>
>>> On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com>
>>> wrote:
>>> > IIUC, the reason for that PR is that they found the string comparison
>>> to
>>> > increase the size in large shuffles. Maybe we should add the ability to
>>> > support the short name to Spark 1.6.2?
>>>
>>> Is that something that really yields noticeable gains in performance?
>>>
>>> If it is, it seems like it would be simple to allow executors register
>>> with the full class name, and map the long names to short names in the
>>> shuffle service itself.
>>>
>>> You could even get fancy and have different ExecutorShuffleInfo
>>> implementations for each shuffle service, with an abstract
>>> "getBlockData" method that gets called instead of the current if/else
>>> in ExternalShuffleBlockResolver.java.
>>>
>>>
>>> --
>>> Marcelo
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN Shuffle service and its compatibility
Posted by Reynold Xin <rx...@databricks.com>.
I talked to Lianhui offline and he said it is not that big of a deal to
revert the patch.
On Tue, Apr 19, 2016 at 9:52 AM, Mark Grover <ma...@apache.org> wrote:
> Thanks.
>
> I'm more than happy to wait for more people to chime in here but I do feel
> that most of us are leaning towards Option B anyways. So, I created a JIRA
> (SPARK-14731) for reverting SPARK-12130 in Spark 2.0 and file a PR shortly.
> Mark
>
> On Tue, Apr 19, 2016 at 7:44 AM, Tom Graves <tg...@yahoo.com.invalid>
> wrote:
>
>> It would be nice if we could keep this compatible between 1.6 and 2.0 so
>> I'm more for Option B at this point since the change made seems minor
>> and we can change to have shuffle service do internally like Marcelo
>> mention. Then lets try to keep compatible, but if there is a forcing
>> function lets figure out a good way to run 2 at once.
>>
>>
>> Tom
>>
>>
>> On Monday, April 18, 2016 5:23 PM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>>
>>
>> On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com> wrote:
>> > IIUC, the reason for that PR is that they found the string comparison to
>> > increase the size in large shuffles. Maybe we should add the ability to
>> > support the short name to Spark 1.6.2?
>>
>> Is that something that really yields noticeable gains in performance?
>>
>> If it is, it seems like it would be simple to allow executors register
>> with the full class name, and map the long names to short names in the
>> shuffle service itself.
>>
>> You could even get fancy and have different ExecutorShuffleInfo
>> implementations for each shuffle service, with an abstract
>> "getBlockData" method that gets called instead of the current if/else
>> in ExternalShuffleBlockResolver.java.
>>
>>
>> --
>> Marcelo
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>>
>>
>>
>
Re: YARN Shuffle service and its compatibility
Posted by Mark Grover <ma...@apache.org>.
Thanks.
I'm more than happy to wait for more people to chime in here but I do feel
that most of us are leaning towards Option B anyways. So, I created a JIRA
(SPARK-14731) for reverting SPARK-12130 in Spark 2.0 and file a PR shortly.
Mark
On Tue, Apr 19, 2016 at 7:44 AM, Tom Graves <tg...@yahoo.com.invalid>
wrote:
> It would be nice if we could keep this compatible between 1.6 and 2.0 so
> I'm more for Option B at this point since the change made seems minor and
> we can change to have shuffle service do internally like Marcelo mention.
> Then lets try to keep compatible, but if there is a forcing function lets
> figure out a good way to run 2 at once.
>
>
> Tom
>
>
> On Monday, April 18, 2016 5:23 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>
> On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com> wrote:
> > IIUC, the reason for that PR is that they found the string comparison to
> > increase the size in large shuffles. Maybe we should add the ability to
> > support the short name to Spark 1.6.2?
>
> Is that something that really yields noticeable gains in performance?
>
> If it is, it seems like it would be simple to allow executors register
> with the full class name, and map the long names to short names in the
> shuffle service itself.
>
> You could even get fancy and have different ExecutorShuffleInfo
> implementations for each shuffle service, with an abstract
> "getBlockData" method that gets called instead of the current if/else
> in ExternalShuffleBlockResolver.java.
>
>
> --
> Marcelo
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
>
>
>
Re: YARN Shuffle service and its compatibility
Posted by Tom Graves <tg...@yahoo.com.INVALID>.
It would be nice if we could keep this compatible between 1.6 and 2.0 so I'm more for Option B at this point since the change made seems minor and we can change to have shuffle service do internally like Marcelo mention. Then lets try to keep compatible, but if there is a forcing function lets figure out a good way to run 2 at once.
Tom
On Monday, April 18, 2016 5:23 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com> wrote:
> IIUC, the reason for that PR is that they found the string comparison to
> increase the size in large shuffles. Maybe we should add the ability to
> support the short name to Spark 1.6.2?
Is that something that really yields noticeable gains in performance?
If it is, it seems like it would be simple to allow executors register
with the full class name, and map the long names to short names in the
shuffle service itself.
You could even get fancy and have different ExecutorShuffleInfo
implementations for each shuffle service, with an abstract
"getBlockData" method that gets called instead of the current if/else
in ExternalShuffleBlockResolver.java.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: YARN Shuffle service and its compatibility
Posted by Mark Grover <ma...@apache.org>.
Thanks for responding, Reynold, Marcelo and Marcin.
>And I think that's really what Mark is proposing. Basically, "don't
>intentionally break backwards compatibility unless it's really
>required" (e.g. SPARK-12130). That would allow option B to work.
Yeah, that's exactly what Option B is proposing.
I also don't think it'd make a huge difference to go back to full class
name but I have explicitly added Lianhui to this thread, who worked on
SPARK-12130, so he can correct me if I am blantantly wrong.
And, even then, we could keep the Spark1 and Spark2 shuffle services
compatible by doing mapping of short-long names or Abstract getBlockData
implementation, if we decide it's necessary.
Mark
On Mon, Apr 18, 2016 at 3:23 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com> wrote:
> > IIUC, the reason for that PR is that they found the string comparison to
> > increase the size in large shuffles. Maybe we should add the ability to
> > support the short name to Spark 1.6.2?
>
> Is that something that really yields noticeable gains in performance?
>
> If it is, it seems like it would be simple to allow executors register
> with the full class name, and map the long names to short names in the
> shuffle service itself.
>
> You could even get fancy and have different ExecutorShuffleInfo
> implementations for each shuffle service, with an abstract
> "getBlockData" method that gets called instead of the current if/else
> in ExternalShuffleBlockResolver.java.
>
> --
> Marcelo
>
Re: YARN Shuffle service and its compatibility
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Mon, Apr 18, 2016 at 3:09 PM, Reynold Xin <rx...@databricks.com> wrote:
> IIUC, the reason for that PR is that they found the string comparison to
> increase the size in large shuffles. Maybe we should add the ability to
> support the short name to Spark 1.6.2?
Is that something that really yields noticeable gains in performance?
If it is, it seems like it would be simple to allow executors register
with the full class name, and map the long names to short names in the
shuffle service itself.
You could even get fancy and have different ExecutorShuffleInfo
implementations for each shuffle service, with an abstract
"getBlockData" method that gets called instead of the current if/else
in ExternalShuffleBlockResolver.java.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: YARN Shuffle service and its compatibility
Posted by Reynold Xin <rx...@databricks.com>.
Got it. So Mark is pushing for "best-effort" support.
IIUC, the reason for that PR is that they found the string comparison to
increase the size in large shuffles. Maybe we should add the ability to
support the short name to Spark 1.6.2?
On Mon, Apr 18, 2016 at 3:05 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <rx...@databricks.com> wrote:
> > The bigger problem is that it is much easier to maintain backward
> > compatibility rather than dictating forward compatibility. For example,
> as
> > Marcin said, if we come up with a slightly different shuffle layout to
> > improve shuffle performance, we wouldn't be able to do that if we want to
> > allow Spark 1.6 shuffle service to read something generated by Spark 2.1.
>
> And I think that's really what Mark is proposing. Basically, "don't
> intentionally break backwards compatibility unless it's really
> required" (e.g. SPARK-12130). That would allow option B to work.
>
> If a new shuffle manager is created, then neither option A nor option
> B would really work. Moving all the shuffle-related classes to a
> different package, to support option A, would be really messy. At that
> point, you're better off maintaining the new shuffle service outside
> of YARN, which is rather messy too.
>
> The best would be if the shuffle service didn't really need to
> understand the shuffle manager, and could find files regardless; I'm
> not sure how feasible that is, though.
>
> --
> Marcelo
>
Re: YARN Shuffle service and its compatibility
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin <rx...@databricks.com> wrote:
> The bigger problem is that it is much easier to maintain backward
> compatibility rather than dictating forward compatibility. For example, as
> Marcin said, if we come up with a slightly different shuffle layout to
> improve shuffle performance, we wouldn't be able to do that if we want to
> allow Spark 1.6 shuffle service to read something generated by Spark 2.1.
And I think that's really what Mark is proposing. Basically, "don't
intentionally break backwards compatibility unless it's really
required" (e.g. SPARK-12130). That would allow option B to work.
If a new shuffle manager is created, then neither option A nor option
B would really work. Moving all the shuffle-related classes to a
different package, to support option A, would be really messy. At that
point, you're better off maintaining the new shuffle service outside
of YARN, which is rather messy too.
The best would be if the shuffle service didn't really need to
understand the shuffle manager, and could find files regardless; I'm
not sure how feasible that is, though.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: YARN Shuffle service and its compatibility
Posted by Reynold Xin <rx...@databricks.com>.
Yea I re-read the email again. It'd work in this case.
The bigger problem is that it is much easier to maintain backward
compatibility rather than dictating forward compatibility. For example, as
Marcin said, if we come up with a slightly different shuffle layout to
improve shuffle performance, we wouldn't be able to do that if we want to
allow Spark 1.6 shuffle service to read something generated by Spark 2.1.
On Mon, Apr 18, 2016 at 1:59 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Mon, Apr 18, 2016 at 1:53 PM, Reynold Xin <rx...@databricks.com> wrote:
> > That's not the only one. For example, the hash shuffle manager has been
> off
> > by default since Spark 1.2, and we'd like to remove it in 2.0:
> > https://github.com/apache/spark/pull/12423
>
> If I understand things correctly, Mark's option B (running a single
> shuffle service, the one from the older Spark release) would still
> work, wouldn't it?
>
> You'd run into problems when Spark adds a new shuffle manager that is
> not known to the old shuffle service, though. Perhaps at that time we
> should investigate making the shuffle service more agnostic to the
> app's shuffle manager.
>
> --
> Marcelo
>
Re: YARN Shuffle service and its compatibility
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Mon, Apr 18, 2016 at 1:53 PM, Reynold Xin <rx...@databricks.com> wrote:
> That's not the only one. For example, the hash shuffle manager has been off
> by default since Spark 1.2, and we'd like to remove it in 2.0:
> https://github.com/apache/spark/pull/12423
If I understand things correctly, Mark's option B (running a single
shuffle service, the one from the older Spark release) would still
work, wouldn't it?
You'd run into problems when Spark adds a new shuffle manager that is
not known to the old shuffle service, though. Perhaps at that time we
should investigate making the shuffle service more agnostic to the
app's shuffle manager.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: YARN Shuffle service and its compatibility
Posted by Reynold Xin <rx...@databricks.com>.
That's not the only one. For example, the hash shuffle manager has been off
by default since Spark 1.2, and we'd like to remove it in 2.0:
https://github.com/apache/spark/pull/12423
How difficult it is to just change the package name to say v2?
On Mon, Apr 18, 2016 at 1:51 PM, Mark Grover <ma...@apache.org> wrote:
> Hi all,
> If you don't use Spark on YARN, you probably don't need to read further.
>
> Here's the *user scenario*:
> There are going to be folks who may be interested in running two versions
> of Spark (say Spark 1.6.x and Spark 2.x) on the same YARN cluster.
>
> And, here's the *problem*:
> That's all fine, should work well. However, there's one problem that
> relates to the YARN shuffle service
> <https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java>.
> This service is run by the YARN Node Managers on all nodes of the cluster
> that have YARN NMs as an auxillary service
> <https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html>
> .
>
> The key question here is -
> Option A: Should the user be running 2 shuffle services - one for Spark
> 1.6.x and one for Spark 2.x?
> OR
> Option B: Should the user be running only 1 shuffle service that services
> both the Spark 1.6.x and Spark 2.x installs? This will likely have to be
> the Spark 1.6.x shuffle service (while ensuring it's forward compatible
> with Spark 2.x).
>
> *Discussion of above options:*
> A few things to note about the shuffle service:
> 1. Looking at the commit history, there aren't a whole of lot of changes
> that go into the shuffle service, rarely ones that are incompatible.
> There's only one incompatible change
> <https://issues.apache.org/jira/browse/SPARK-12130> that's been made to
> the shuffle service, as far as I can tell, and that too, seems fairly
> cosmetic.
> 2. Shuffle services for 1.6.x and 2.x serve very similar purpose (to
> provide shuffle blocks) and can easily be just one service that does it,
> even on a YARN cluster that runs both Spark 1.x and Spark 2.x.
> 3. The shuffle service is not version-spaced. This means that, the way the
> code is currently, if we were to drop the jars for Spark1 and Spark2's
> shuffle service in YARN NM's classpath, YARN NM won't be able to start both
> services. It would arbitrarily pick one service to start (based on what
> appears on the classpath first). Also, the service name is hardcoded
> <https://github.com/apache/spark/blob/master/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java#L96>
> in Spark code and that name is also not version-spaced.
>
> Option A is arguably cleaner but it's more operational overhead and some
> code relocation/shading/version-spacing/name-spacing to make it work (due
> to #3 above), potentially to not a whole lot of value (given #2 above).
>
> Option B is simpler, lean and more operationally efficient. However, that
> requires that we as a community, keep Spark 1's shuffle service forward
> compatible with Spark 2 i.e. don't break compatibility between Spark1's and
> Spark2's shuffle service. We could even add a test (mima?) to assert that
> during the life time of Spark2. If we do go down that way, we should revert
> SPARK-12130 <https://issues.apache.org/jira/browse/SPARK-12130> - the
> only backwards incompatible change made to Spark2 shuffle service so far.
>
> My personal vote goes towards Option B and I think reverting SPARK-12130
> is ok. What do others think?
>
> Thanks!
> Mark
>
>