You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/06/13 17:50:46 UTC
[YARN] Small fix for yarn.Client to use buildPath (not Path.SEPARATOR)
Hi,
Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
[1] to build a CLASSPATH entry while another similar-looking line uses
buildPath method [2].
Could a pull request with a change to use buildPath at [1] be
accepted? I'm always confused how to fix such small changes.
[1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
[2] Path.SEPARATOR
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [YARN] Small fix for yarn.Client to use buildPath (not
Path.SEPARATOR)
Posted by Steve Loughran <st...@hortonworks.com>.
> On 14 Jun 2016, at 13:58, Jacek Laskowski <ja...@japila.pl> wrote:
>
> Hi Steve and Sean,
>
> Didn't expect such a warm welcome from Sean and you! Since I'm with
> Spark on YARN these days, let me see what I can do to make it nicer.
> Thanks!
>
> I'm going to change Spark to use buildPath first. And then propose
> another patch to use Environment.CLASS_PATH_SEPARATOR instead. And
> only then I could work on
> https://issues.apache.org/jira/browse/YARN-5247. Is this about
> changing the annotation(s) only?
>
yes; though I should warn that I have evidence that some people have been breaking things tagged a stable;
https://issues.apache.org/jira/browse/YARN-5130
what that @Stable marker does do is give you ability to complain when things break
> Thanks for your support!
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Jun 14, 2016 at 1:44 PM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> if you want to be able to build up CPs on windows to run on a Linux cluster, or vice-versa, you really need to be using the Environment.CLASS_PATH_SEPARATOR field, "<CPS>". This is expanded in the cluster, not in the client
>>
>> Although tagged as @Public, @Unstable, it's been in there sinceYARN-1824 & Hadoop 2.4; things rely on it. If someone wants to fix that by submitting a patch to YARN-5247; I'll review it.
>>
>>> On 13 Jun 2016, at 20:06, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> Yeah it does the same thing anyway. It's fine to consistently use the
>>> method. I think there's an instance in ClientSuite that can use it.
>>>
>>> On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>>> Hi,
>>>>
>>>> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
>>>> [1] to build a CLASSPATH entry while another similar-looking line uses
>>>> buildPath method [2].
>>>>
>>>> Could a pull request with a change to use buildPath at [1] be
>>>> accepted? I'm always confused how to fix such small changes.
>>>>
>>>> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
>>>> [2] Path.SEPARATOR
>>>>
>>>> Pozdrawiam,
>>>> Jacek Laskowski
>>>> ----
>>>> https://medium.com/@jaceklaskowski/
>>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [YARN] Small fix for yarn.Client to use buildPath (not Path.SEPARATOR)
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Steve and Sean,
Didn't expect such a warm welcome from Sean and you! Since I'm with
Spark on YARN these days, let me see what I can do to make it nicer.
Thanks!
I'm going to change Spark to use buildPath first. And then propose
another patch to use Environment.CLASS_PATH_SEPARATOR instead. And
only then I could work on
https://issues.apache.org/jira/browse/YARN-5247. Is this about
changing the annotation(s) only?
Thanks for your support!
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jun 14, 2016 at 1:44 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
> if you want to be able to build up CPs on windows to run on a Linux cluster, or vice-versa, you really need to be using the Environment.CLASS_PATH_SEPARATOR field, "<CPS>". This is expanded in the cluster, not in the client
>
> Although tagged as @Public, @Unstable, it's been in there sinceYARN-1824 & Hadoop 2.4; things rely on it. If someone wants to fix that by submitting a patch to YARN-5247; I'll review it.
>
>> On 13 Jun 2016, at 20:06, Sean Owen <so...@cloudera.com> wrote:
>>
>> Yeah it does the same thing anyway. It's fine to consistently use the
>> method. I think there's an instance in ClientSuite that can use it.
>>
>> On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>>> Hi,
>>>
>>> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
>>> [1] to build a CLASSPATH entry while another similar-looking line uses
>>> buildPath method [2].
>>>
>>> Could a pull request with a change to use buildPath at [1] be
>>> accepted? I'm always confused how to fix such small changes.
>>>
>>> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
>>> [2] Path.SEPARATOR
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [YARN] Small fix for yarn.Client to use buildPath (not
Path.SEPARATOR)
Posted by Steve Loughran <st...@hortonworks.com>.
if you want to be able to build up CPs on windows to run on a Linux cluster, or vice-versa, you really need to be using the Environment.CLASS_PATH_SEPARATOR field, "<CPS>". This is expanded in the cluster, not in the client
Although tagged as @Public, @Unstable, it's been in there sinceYARN-1824 & Hadoop 2.4; things rely on it. If someone wants to fix that by submitting a patch to YARN-5247; I'll review it.
> On 13 Jun 2016, at 20:06, Sean Owen <so...@cloudera.com> wrote:
>
> Yeah it does the same thing anyway. It's fine to consistently use the
> method. I think there's an instance in ClientSuite that can use it.
>
> On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>> Hi,
>>
>> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
>> [1] to build a CLASSPATH entry while another similar-looking line uses
>> buildPath method [2].
>>
>> Could a pull request with a change to use buildPath at [1] be
>> accepted? I'm always confused how to fix such small changes.
>>
>> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
>> [2] Path.SEPARATOR
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [YARN] Small fix for yarn.Client to use buildPath (not Path.SEPARATOR)
Posted by Sean Owen <so...@cloudera.com>.
Yeah it does the same thing anyway. It's fine to consistently use the
method. I think there's an instance in ClientSuite that can use it.
On Mon, Jun 13, 2016 at 6:50 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> Just noticed that yarn.Client#populateClasspath uses Path.SEPARATOR
> [1] to build a CLASSPATH entry while another similar-looking line uses
> buildPath method [2].
>
> Could a pull request with a change to use buildPath at [1] be
> accepted? I'm always confused how to fix such small changes.
>
> [1] https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1298
> [2] Path.SEPARATOR
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org