You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mridul Muralidharan <mr...@gmail.com> on 2014/04/23 15:07:21 UTC
Re: [jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on
command line
This breaks all existing jobs which are not using spark-submit.
The consensus was not to break compatibility unless there was an overriding
reason to do so
On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164]
>
> Thomas Graves commented on SPARK-1576:
> --------------------------------------
>
> Is this meant for the driver or the executors? The spark-submit script
> has a command line option for the driver: --driver-java-options.
> I believe the intent of https://github.com/apache/spark/pull/299 was to
> not expose SPARK_JAVA_OPTS to the user anymore.
>
> > Passing of JAVA_OPTS to YARN on command line
> > --------------------------------------------
> >
> > Key: SPARK-1576
> > URL: https://issues.apache.org/jira/browse/SPARK-1576
> > Project: Spark
> > Issue Type: Improvement
> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
> > Reporter: Nishkam Ravi
> > Fix For: 0.9.0, 1.0.0, 0.9.1
> >
> > Attachments: SPARK-1576.patch
> >
> >
> > JAVA_OPTS can be passed by using either env variables (i.e.,
> SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change). It
> would be good to allow the user to pass them on command line as well to
> restrict scope to single application invocation.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.2#6252)
>
Re: [jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line
Posted by Tom Graves <tg...@yahoo.com>.
can you be more specific? What breaks existing jobs? If you are referring to my comment, SPARK_JAVA_OPTS still works but I think the intent is to move away from it.
Tom
On Wednesday, April 23, 2014 8:07 AM, Mridul Muralidharan <mr...@gmail.com> wrote:
This breaks all existing jobs which are not using spark-submit.
The consensus was not to break compatibility unless there was an overriding
reason to do so
On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164]
>
> Thomas Graves commented on SPARK-1576:
> --------------------------------------
>
> Is this meant for the driver or the executors? The spark-submit script
> has a command line option for the driver: --driver-java-options.
> I believe the intent of https://github.com/apache/spark/pull/299 was to
> not expose SPARK_JAVA_OPTS to the user anymore.
>
> > Passing of JAVA_OPTS to YARN on command line
> > --------------------------------------------
> >
> > Key: SPARK-1576
> > URL: https://issues.apache.org/jira/browse/SPARK-1576
> > Project: Spark
> > Issue Type: Improvement
> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
> > Reporter: Nishkam Ravi
> > Fix For: 0.9.0, 1.0.0, 0.9.1
> >
> > Attachments: SPARK-1576.patch
> >
> >
> > JAVA_OPTS can be passed by using either env variables (i.e.,
> SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change). It
> would be good to allow the user to pass them on command line as well to
> restrict scope to single application invocation.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.2#6252)
>
Re: [jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on
command line
Posted by Nishkam Ravi <nr...@cloudera.com>.
It would probably be best to retain support for SPARK_JAVA_OPTS in
ClientBase though..for developers that may have been using it.
On Wed, Apr 23, 2014 at 6:26 PM, Nishkam Ravi <nr...@cloudera.com> wrote:
> Bit of a race condition here it seems. Patrick made a few changes
> yesterday around the same time as I did (in ClientBase.scala):
>
> for ((k, v) <- sys.props.filterKeys(_.startsWith("spark")))
> { JAVA_OPTS += "-D" + k + "=" + "\\\"" + v + "\\\"" }
>
> This would allow JAVA_OPTS to be passed on the command line to the
> ApplicationMaster, and accomplishes the same things as creation of a new
> command line flag --spark-java-opts.
>
>
> Mridul, the use of SPARK_JAVA_OPTS has been intentionally suppressed.
>
>
> On Wed, Apr 23, 2014 at 10:54 AM, Mridul Muralidharan <mr...@gmail.com>wrote:
>
>> Sorry, I misread - I meant SPARK_JAVA_OPTS - not JAVA_OPTS.
>> See here : https://issues.apache.org/jira/browse/SPARK-1588
>>
>> Regards,
>> Mridul
>>
>> On Wed, Apr 23, 2014 at 6:37 PM, Mridul Muralidharan <mr...@gmail.com>
>> wrote:
>> > This breaks all existing jobs which are not using spark-submit.
>> > The consensus was not to break compatibility unless there was an
>> overriding
>> > reason to do so
>> >
>> > On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <ji...@apache.org>
>> wrote:
>> >>
>> >>
>> >> [
>> >>
>> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164
>> >> ]
>> >>
>> >> Thomas Graves commented on SPARK-1576:
>> >> --------------------------------------
>> >>
>> >> Is this meant for the driver or the executors? The spark-submit script
>> >> has a command line option for the driver: --driver-java-options.
>> >> I believe the intent of https://github.com/apache/spark/pull/299 was
>> to
>> >> not expose SPARK_JAVA_OPTS to the user anymore.
>> >>
>> >> > Passing of JAVA_OPTS to YARN on command line
>> >> > --------------------------------------------
>> >> >
>> >> > Key: SPARK-1576
>> >> > URL:
>> https://issues.apache.org/jira/browse/SPARK-1576
>> >> > Project: Spark
>> >> > Issue Type: Improvement
>> >> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
>> >> > Reporter: Nishkam Ravi
>> >> > Fix For: 0.9.0, 1.0.0, 0.9.1
>> >> >
>> >> > Attachments: SPARK-1576.patch
>> >> >
>> >> >
>> >> > JAVA_OPTS can be passed by using either env variables (i.e.,
>> >> > SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change).
>> It would
>> >> > be good to allow the user to pass them on command line as well to
>> restrict
>> >> > scope to single application invocation.
>> >>
>> >>
>> >>
>> >> --
>> >> This message was sent by Atlassian JIRA
>> >> (v6.2#6252)
>>
>
>
Re: [jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on
command line
Posted by Nishkam Ravi <nr...@cloudera.com>.
Bit of a race condition here it seems. Patrick made a few changes yesterday
around the same time as I did (in ClientBase.scala):
for ((k, v) <- sys.props.filterKeys(_.startsWith("spark")))
{ JAVA_OPTS += "-D" + k + "=" + "\\\"" + v + "\\\"" }
This would allow JAVA_OPTS to be passed on the command line to the
ApplicationMaster, and accomplishes the same things as creation of a new
command line flag --spark-java-opts.
Mridul, the use of SPARK_JAVA_OPTS has been intentionally suppressed.
On Wed, Apr 23, 2014 at 10:54 AM, Mridul Muralidharan <mr...@gmail.com>wrote:
> Sorry, I misread - I meant SPARK_JAVA_OPTS - not JAVA_OPTS.
> See here : https://issues.apache.org/jira/browse/SPARK-1588
>
> Regards,
> Mridul
>
> On Wed, Apr 23, 2014 at 6:37 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
> > This breaks all existing jobs which are not using spark-submit.
> > The consensus was not to break compatibility unless there was an
> overriding
> > reason to do so
> >
> > On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <ji...@apache.org> wrote:
> >>
> >>
> >> [
> >>
> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164
> >> ]
> >>
> >> Thomas Graves commented on SPARK-1576:
> >> --------------------------------------
> >>
> >> Is this meant for the driver or the executors? The spark-submit script
> >> has a command line option for the driver: --driver-java-options.
> >> I believe the intent of https://github.com/apache/spark/pull/299 was to
> >> not expose SPARK_JAVA_OPTS to the user anymore.
> >>
> >> > Passing of JAVA_OPTS to YARN on command line
> >> > --------------------------------------------
> >> >
> >> > Key: SPARK-1576
> >> > URL: https://issues.apache.org/jira/browse/SPARK-1576
> >> > Project: Spark
> >> > Issue Type: Improvement
> >> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
> >> > Reporter: Nishkam Ravi
> >> > Fix For: 0.9.0, 1.0.0, 0.9.1
> >> >
> >> > Attachments: SPARK-1576.patch
> >> >
> >> >
> >> > JAVA_OPTS can be passed by using either env variables (i.e.,
> >> > SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change).
> It would
> >> > be good to allow the user to pass them on command line as well to
> restrict
> >> > scope to single application invocation.
> >>
> >>
> >>
> >> --
> >> This message was sent by Atlassian JIRA
> >> (v6.2#6252)
>
Re: [jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on
command line
Posted by Mridul Muralidharan <mr...@gmail.com>.
Sorry, I misread - I meant SPARK_JAVA_OPTS - not JAVA_OPTS.
See here : https://issues.apache.org/jira/browse/SPARK-1588
Regards,
Mridul
On Wed, Apr 23, 2014 at 6:37 PM, Mridul Muralidharan <mr...@gmail.com> wrote:
> This breaks all existing jobs which are not using spark-submit.
> The consensus was not to break compatibility unless there was an overriding
> reason to do so
>
> On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <ji...@apache.org> wrote:
>>
>>
>> [
>> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164
>> ]
>>
>> Thomas Graves commented on SPARK-1576:
>> --------------------------------------
>>
>> Is this meant for the driver or the executors? The spark-submit script
>> has a command line option for the driver: --driver-java-options.
>> I believe the intent of https://github.com/apache/spark/pull/299 was to
>> not expose SPARK_JAVA_OPTS to the user anymore.
>>
>> > Passing of JAVA_OPTS to YARN on command line
>> > --------------------------------------------
>> >
>> > Key: SPARK-1576
>> > URL: https://issues.apache.org/jira/browse/SPARK-1576
>> > Project: Spark
>> > Issue Type: Improvement
>> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
>> > Reporter: Nishkam Ravi
>> > Fix For: 0.9.0, 1.0.0, 0.9.1
>> >
>> > Attachments: SPARK-1576.patch
>> >
>> >
>> > JAVA_OPTS can be passed by using either env variables (i.e.,
>> > SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change). It would
>> > be good to allow the user to pass them on command line as well to restrict
>> > scope to single application invocation.
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.2#6252)