You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Evan Chan <ev...@ooyala.com> on 2013/09/29 00:23:28 UTC
spark.hostPort
Hey guys,
Does anyone see a reason to keep the "spark.hostPort" system property
around?
It is cleared by lots of tests, set in two places (SparkEnv, and
StandaloneExecutorBackend),
and only used as follows:
Utils.localHostPort() reads it
localHostPort is called from
- BlockManager (which calls it, but _never_ uses the resulting output!)
- MapOutputTracker (which passes it as a value, not sure to where)
Just trying to clean house on properties.
thanks,
Evan
--
--
Evan Chan
Staff Engineer
ev@ooyala.com |
<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: spark.hostPort
Posted by Matei Zaharia <ma...@gmail.com>.
Sure, that makes sense.
Matei
On Sep 29, 2013, at 1:51 AM, Evan Chan <ev...@ooyala.com> wrote:
> Got it. How about we make it such that any property spark.akka.* will be
> forwarded to akka as akka.* (ie with the "spark." part stripped). So
> there is no need to manually transcribe properties over.
>
>
> On Sat, Sep 28, 2013 at 4:53 PM, Matei Zaharia <ma...@gmail.com>wrote:
>
>> The main reason I wanted them separate was that people might use Akka in
>> their own application for other things. As such, the Spark Akka properties
>> only affect Akka instances started by Spark. I think we should keep them
>> separate to avoid messing with users' applications.
>>
>> Matei
>>
>> On Sep 28, 2013, at 7:28 PM, Evan Chan <ev...@ooyala.com> wrote:
>>
>>> On a related note, there are tons of spark.akka.* properties.
>>>
>>> Does anyone see why we should not just use the base akka.* properties
>>> defined by Akka itself? Right now the code simply maps spark.akka.* to
>>> akka.* when starting Akka.
>>>
>>> The only reasons I see for keeping spark.akka is that we might not want
>>> people fiddling with the base Akka properties. On the other hand, Akka's
>>> documentation is excellent, and using akka's own config properties means
>> we
>>> no longer need to keep up with this shim layer.
>>>
>>> -Evan
>>>
>>>
>>>
>>> On Sat, Sep 28, 2013 at 3:44 PM, Matei Zaharia <matei.zaharia@gmail.com
>>> wrote:
>>>
>>>> Hi Evan,
>>>>
>>>> I think this is an old property that isn't used anymore, so it would be
>>>> good to clean it up and get rid of it.
>>>>
>>>> Matei
>>>>
>>>> On Sep 28, 2013, at 6:23 PM, Evan Chan <ev...@ooyala.com> wrote:
>>>>
>>>>> Hey guys,
>>>>>
>>>>> Does anyone see a reason to keep the "spark.hostPort" system property
>>>>> around?
>>>>>
>>>>> It is cleared by lots of tests, set in two places (SparkEnv, and
>>>>> StandaloneExecutorBackend),
>>>>> and only used as follows:
>>>>>
>>>>> Utils.localHostPort() reads it
>>>>> localHostPort is called from
>>>>> - BlockManager (which calls it, but _never_ uses the resulting output!)
>>>>> - MapOutputTracker (which passes it as a value, not sure to where)
>>>>>
>>>>> Just trying to clean house on properties.
>>>>>
>>>>> thanks,
>>>>> Evan
>>>>>
>>>>> --
>>>>> --
>>>>> Evan Chan
>>>>> Staff Engineer
>>>>> ev@ooyala.com |
>>>>>
>>>>> <http://www.ooyala.com/>
>>>>> <http://www.facebook.com/ooyala><
>> http://www.linkedin.com/company/ooyala
>>>>> <http://www.twitter.com/ooyala>
>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Evan Chan
>>> Staff Engineer
>>> ev@ooyala.com |
>>>
>>> <http://www.ooyala.com/>
>>> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala
>>> <http://www.twitter.com/ooyala>
>>
>>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: spark.hostPort
Posted by Evan Chan <ev...@ooyala.com>.
Got it. How about we make it such that any property spark.akka.* will be
forwarded to akka as akka.* (ie with the "spark." part stripped). So
there is no need to manually transcribe properties over.
On Sat, Sep 28, 2013 at 4:53 PM, Matei Zaharia <ma...@gmail.com>wrote:
> The main reason I wanted them separate was that people might use Akka in
> their own application for other things. As such, the Spark Akka properties
> only affect Akka instances started by Spark. I think we should keep them
> separate to avoid messing with users' applications.
>
> Matei
>
> On Sep 28, 2013, at 7:28 PM, Evan Chan <ev...@ooyala.com> wrote:
>
> > On a related note, there are tons of spark.akka.* properties.
> >
> > Does anyone see why we should not just use the base akka.* properties
> > defined by Akka itself? Right now the code simply maps spark.akka.* to
> > akka.* when starting Akka.
> >
> > The only reasons I see for keeping spark.akka is that we might not want
> > people fiddling with the base Akka properties. On the other hand, Akka's
> > documentation is excellent, and using akka's own config properties means
> we
> > no longer need to keep up with this shim layer.
> >
> > -Evan
> >
> >
> >
> > On Sat, Sep 28, 2013 at 3:44 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
> >
> >> Hi Evan,
> >>
> >> I think this is an old property that isn't used anymore, so it would be
> >> good to clean it up and get rid of it.
> >>
> >> Matei
> >>
> >> On Sep 28, 2013, at 6:23 PM, Evan Chan <ev...@ooyala.com> wrote:
> >>
> >>> Hey guys,
> >>>
> >>> Does anyone see a reason to keep the "spark.hostPort" system property
> >>> around?
> >>>
> >>> It is cleared by lots of tests, set in two places (SparkEnv, and
> >>> StandaloneExecutorBackend),
> >>> and only used as follows:
> >>>
> >>> Utils.localHostPort() reads it
> >>> localHostPort is called from
> >>> - BlockManager (which calls it, but _never_ uses the resulting output!)
> >>> - MapOutputTracker (which passes it as a value, not sure to where)
> >>>
> >>> Just trying to clean house on properties.
> >>>
> >>> thanks,
> >>> Evan
> >>>
> >>> --
> >>> --
> >>> Evan Chan
> >>> Staff Engineer
> >>> ev@ooyala.com |
> >>>
> >>> <http://www.ooyala.com/>
> >>> <http://www.facebook.com/ooyala><
> http://www.linkedin.com/company/ooyala
> >>> <http://www.twitter.com/ooyala>
> >>
> >>
> >
> >
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > ev@ooyala.com |
> >
> > <http://www.ooyala.com/>
> > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala
> ><http://www.twitter.com/ooyala>
>
>
--
--
Evan Chan
Staff Engineer
ev@ooyala.com |
<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: spark.hostPort
Posted by Matei Zaharia <ma...@gmail.com>.
The main reason I wanted them separate was that people might use Akka in their own application for other things. As such, the Spark Akka properties only affect Akka instances started by Spark. I think we should keep them separate to avoid messing with users' applications.
Matei
On Sep 28, 2013, at 7:28 PM, Evan Chan <ev...@ooyala.com> wrote:
> On a related note, there are tons of spark.akka.* properties.
>
> Does anyone see why we should not just use the base akka.* properties
> defined by Akka itself? Right now the code simply maps spark.akka.* to
> akka.* when starting Akka.
>
> The only reasons I see for keeping spark.akka is that we might not want
> people fiddling with the base Akka properties. On the other hand, Akka's
> documentation is excellent, and using akka's own config properties means we
> no longer need to keep up with this shim layer.
>
> -Evan
>
>
>
> On Sat, Sep 28, 2013 at 3:44 PM, Matei Zaharia <ma...@gmail.com>wrote:
>
>> Hi Evan,
>>
>> I think this is an old property that isn't used anymore, so it would be
>> good to clean it up and get rid of it.
>>
>> Matei
>>
>> On Sep 28, 2013, at 6:23 PM, Evan Chan <ev...@ooyala.com> wrote:
>>
>>> Hey guys,
>>>
>>> Does anyone see a reason to keep the "spark.hostPort" system property
>>> around?
>>>
>>> It is cleared by lots of tests, set in two places (SparkEnv, and
>>> StandaloneExecutorBackend),
>>> and only used as follows:
>>>
>>> Utils.localHostPort() reads it
>>> localHostPort is called from
>>> - BlockManager (which calls it, but _never_ uses the resulting output!)
>>> - MapOutputTracker (which passes it as a value, not sure to where)
>>>
>>> Just trying to clean house on properties.
>>>
>>> thanks,
>>> Evan
>>>
>>> --
>>> --
>>> Evan Chan
>>> Staff Engineer
>>> ev@ooyala.com |
>>>
>>> <http://www.ooyala.com/>
>>> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala
>>> <http://www.twitter.com/ooyala>
>>
>>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: spark.hostPort
Posted by Evan Chan <ev...@ooyala.com>.
On a related note, there are tons of spark.akka.* properties.
Does anyone see why we should not just use the base akka.* properties
defined by Akka itself? Right now the code simply maps spark.akka.* to
akka.* when starting Akka.
The only reasons I see for keeping spark.akka is that we might not want
people fiddling with the base Akka properties. On the other hand, Akka's
documentation is excellent, and using akka's own config properties means we
no longer need to keep up with this shim layer.
-Evan
On Sat, Sep 28, 2013 at 3:44 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Hi Evan,
>
> I think this is an old property that isn't used anymore, so it would be
> good to clean it up and get rid of it.
>
> Matei
>
> On Sep 28, 2013, at 6:23 PM, Evan Chan <ev...@ooyala.com> wrote:
>
> > Hey guys,
> >
> > Does anyone see a reason to keep the "spark.hostPort" system property
> > around?
> >
> > It is cleared by lots of tests, set in two places (SparkEnv, and
> > StandaloneExecutorBackend),
> > and only used as follows:
> >
> > Utils.localHostPort() reads it
> > localHostPort is called from
> > - BlockManager (which calls it, but _never_ uses the resulting output!)
> > - MapOutputTracker (which passes it as a value, not sure to where)
> >
> > Just trying to clean house on properties.
> >
> > thanks,
> > Evan
> >
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > ev@ooyala.com |
> >
> > <http://www.ooyala.com/>
> > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala
> ><http://www.twitter.com/ooyala>
>
>
--
--
Evan Chan
Staff Engineer
ev@ooyala.com |
<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Re: spark.hostPort
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Evan,
I think this is an old property that isn't used anymore, so it would be good to clean it up and get rid of it.
Matei
On Sep 28, 2013, at 6:23 PM, Evan Chan <ev...@ooyala.com> wrote:
> Hey guys,
>
> Does anyone see a reason to keep the "spark.hostPort" system property
> around?
>
> It is cleared by lots of tests, set in two places (SparkEnv, and
> StandaloneExecutorBackend),
> and only used as follows:
>
> Utils.localHostPort() reads it
> localHostPort is called from
> - BlockManager (which calls it, but _never_ uses the resulting output!)
> - MapOutputTracker (which passes it as a value, not sure to where)
>
> Just trying to clean house on properties.
>
> thanks,
> Evan
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>