You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/06/26 12:58:49 UTC
Using SHUFFLE_SERVICE_ENABLED for MesosCoarseGrainedSchedulerBackend,
BlockManager, and Utils?
Hi,
I've just noticed that there is the private[spark] val
SHUFFLE_SERVICE_ENABLED in package object config [1]
[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L74-L75
However MesosCoarseGrainedSchedulerBackend [2], BlockManager [3] and
Utils [4] are all using their own copies.
Would that be acceptable* to send a pull request to get rid of this redundancy?
[*] I'm staring at @srowen for his nodding in agreement :-)
[2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L71
[3] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L73-L74
[4] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L748
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using SHUFFLE_SERVICE_ENABLED for MesosCoarseGrainedSchedulerBackend,
BlockManager, and Utils?
Posted by Jacek Laskowski <ja...@japila.pl>.
Thanks Sean. I'm going to create a JIRA for it and start the work under it.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Mon, Jun 27, 2016 at 9:19 AM, Sean Owen <so...@cloudera.com> wrote:
> That seems OK. If it introduces another module dependency we'd have to
> think about it. I assume these constants should really be used
> consistently everywhere if possible, just because it otherwise means
> duplicating the defaults and possibly incorrectly. I think you could
> have a look at that more broadly too.
>
> On Sun, Jun 26, 2016 at 1:58 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>> Hi,
>>
>> I've just noticed that there is the private[spark] val
>> SHUFFLE_SERVICE_ENABLED in package object config [1]
>>
>> [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L74-L75
>>
>> However MesosCoarseGrainedSchedulerBackend [2], BlockManager [3] and
>> Utils [4] are all using their own copies.
>>
>> Would that be acceptable* to send a pull request to get rid of this redundancy?
>>
>> [*] I'm staring at @srowen for his nodding in agreement :-)
>>
>> [2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L71
>> [3] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L73-L74
>> [4] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L748
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using SHUFFLE_SERVICE_ENABLED for MesosCoarseGrainedSchedulerBackend,
BlockManager, and Utils?
Posted by Sean Owen <so...@cloudera.com>.
That seems OK. If it introduces another module dependency we'd have to
think about it. I assume these constants should really be used
consistently everywhere if possible, just because it otherwise means
duplicating the defaults and possibly incorrectly. I think you could
have a look at that more broadly too.
On Sun, Jun 26, 2016 at 1:58 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> I've just noticed that there is the private[spark] val
> SHUFFLE_SERVICE_ENABLED in package object config [1]
>
> [1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/internal/config/package.scala#L74-L75
>
> However MesosCoarseGrainedSchedulerBackend [2], BlockManager [3] and
> Utils [4] are all using their own copies.
>
> Would that be acceptable* to send a pull request to get rid of this redundancy?
>
> [*] I'm staring at @srowen for his nodding in agreement :-)
>
> [2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L71
> [3] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L73-L74
> [4] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L748
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org