You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Reynold Xin <rx...@databricks.com> on 2016/03/16 22:52:27 UTC
[discuss] making SparkEnv private in Spark 2.0
Any objections? Please articulate your use case. SparkEnv is a weird one
because it was documented as "private" but not marked as so in class
visibility.
* NOTE: This is not intended for external use. This is exposed for Shark
and may be made private
* in a future release.
I do see Hive using it to get the config variable. That can probably be
propagated through other means.
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Mridul Muralidharan <mr...@gmail.com>.
We have custom join's that leverage it.
It is used to get to direct shuffle'ed iterator - without needing
sort/aggregate/etc.
IIRC the only way to get to it from ShuffleHandle is via shuffle manager.
Regards,
Mridul
On Wed, Mar 16, 2016 at 3:36 PM, Reynold Xin <rx...@databricks.com> wrote:
>
> On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
>>
>> b) Shuffle manager (to get shuffle reader)
>
>
> What's the use case for shuffle manager/reader? This seems like using super
> internal APIs in applications.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Mridul Muralidharan <mr...@gmail.com>.
We have custom join's that leverage it.
It is used to get to direct shuffle'ed iterator - without needing
sort/aggregate/etc.
IIRC the only way to get to it from ShuffleHandle is via shuffle manager.
Regards,
Mridul
On Wed, Mar 16, 2016 at 3:36 PM, Reynold Xin <rx...@databricks.com> wrote:
>
> On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
>>
>> b) Shuffle manager (to get shuffle reader)
>
>
> What's the use case for shuffle manager/reader? This seems like using super
> internal APIs in applications.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Reynold Xin <rx...@databricks.com>.
On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan <mr...@gmail.com>
wrote:
> b) Shuffle manager (to get shuffle reader)
>
What's the use case for shuffle manager/reader? This seems like using super
internal APIs in applications.
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Reynold Xin <rx...@databricks.com>.
On Wed, Mar 16, 2016 at 3:29 PM, Mridul Muralidharan <mr...@gmail.com>
wrote:
> b) Shuffle manager (to get shuffle reader)
>
What's the use case for shuffle manager/reader? This seems like using super
internal APIs in applications.
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Mridul Muralidharan <mr...@gmail.com>.
We use it in executors to get to :
a) spark conf (for getting to hadoop config in map doing custom
writing of side-files)
b) Shuffle manager (to get shuffle reader)
Not sure if there are alternative ways to get to these.
Regards,
Mridul
On Wed, Mar 16, 2016 at 2:52 PM, Reynold Xin <rx...@databricks.com> wrote:
> Any objections? Please articulate your use case. SparkEnv is a weird one
> because it was documented as "private" but not marked as so in class
> visibility.
>
> * NOTE: This is not intended for external use. This is exposed for Shark
> and may be made private
> * in a future release.
>
>
> I do see Hive using it to get the config variable. That can probably be
> propagated through other means.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [discuss] making SparkEnv private in Spark 2.0
Posted by Mridul Muralidharan <mr...@gmail.com>.
We use it in executors to get to :
a) spark conf (for getting to hadoop config in map doing custom
writing of side-files)
b) Shuffle manager (to get shuffle reader)
Not sure if there are alternative ways to get to these.
Regards,
Mridul
On Wed, Mar 16, 2016 at 2:52 PM, Reynold Xin <rx...@databricks.com> wrote:
> Any objections? Please articulate your use case. SparkEnv is a weird one
> because it was documented as "private" but not marked as so in class
> visibility.
>
> * NOTE: This is not intended for external use. This is exposed for Shark
> and may be made private
> * in a future release.
>
>
> I do see Hive using it to get the config variable. That can probably be
> propagated through other means.
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org