You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jo Voordeckers <jo...@gmail.com> on 2015/11/17 02:46:36 UTC
Mesos cluster dispatcher doesn't respect most args from the submit req
Hi all,
I'm running the mesos cluster dispatcher, however when I submit jobs with
things like jvm args, classpath order and UI port aren't added to the
commandline executed by the mesos scheduler. In fact it only cares about
the class, jar and num cores/mem.
https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
I've made an attempt at adding a few of the args that I believe are useful
to the MesosClusterScheduler class, which seems to solve my problem.
Please have a look:
https://github.com/apache/spark/pull/9752
Thanks
- Jo Voordeckers
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Jo Voordeckers <jo...@gmail.com>.
Hi Tim,
I've done more forensics on this bug, see my comment here:
https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843
- Jo Voordeckers
On Tue, Nov 17, 2015 at 12:01 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Jo,
>
> Thanks for the links, I would expected the properties to be in
> scheduler properties but I need to double check.
>
> I'll be looking into these problems this week.
>
> Tim
>
> On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers
> <jo...@gmail.com> wrote:
> > On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <
> iulian.dragos@typesafe.com>
> > wrote:
> >>
> >> I think it actually tries to send all properties as part of
> >> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
> >>
> >>
> >>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
> >>
> >
> > Aha that's interesting, I overlooked that line, I'll debug some more
> today
> > because I know for sure that those options don't make it onto the
> > commandline when I was running it in my debugger.
> >
> >>
> >> Can you please open a Jira ticket and describe also the symptoms? This
> >> might be related, or the same issue: SPARK-11280 and also SPARK-11327
> >
> >
> > SPARK-11327 is exactly my problem, but I don't run docker.
> >
> > - Jo
> >
> >> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <
> jo.voordeckers@gmail.com>
> >> wrote:
> >>>
> >>>
> >>> Hi all,
> >>>
> >>> I'm running the mesos cluster dispatcher, however when I submit jobs
> with
> >>> things like jvm args, classpath order and UI port aren't added to the
> >>> commandline executed by the mesos scheduler. In fact it only cares
> about the
> >>> class, jar and num cores/mem.
> >>>
> >>>
> >>>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
> >>>
> >>> I've made an attempt at adding a few of the args that I believe are
> >>> useful to the MesosClusterScheduler class, which seems to solve my
> problem.
> >>>
> >>> Please have a look:
> >>>
> >>> https://github.com/apache/spark/pull/9752
> >>>
> >>> Thanks
> >>>
> >>> - Jo Voordeckers
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> --
> >> Iulian Dragos
> >>
> >> ------
> >> Reactive Apps on the JVM
> >> www.typesafe.com
> >>
> >
>
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Jo Voordeckers <jo...@gmail.com>.
Hi Tim,
I've done more forensics on this bug, see my comment here:
https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843
- Jo Voordeckers
On Tue, Nov 17, 2015 at 12:01 PM, Timothy Chen <tn...@gmail.com> wrote:
> Hi Jo,
>
> Thanks for the links, I would expected the properties to be in
> scheduler properties but I need to double check.
>
> I'll be looking into these problems this week.
>
> Tim
>
> On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers
> <jo...@gmail.com> wrote:
> > On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <
> iulian.dragos@typesafe.com>
> > wrote:
> >>
> >> I think it actually tries to send all properties as part of
> >> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
> >>
> >>
> >>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
> >>
> >
> > Aha that's interesting, I overlooked that line, I'll debug some more
> today
> > because I know for sure that those options don't make it onto the
> > commandline when I was running it in my debugger.
> >
> >>
> >> Can you please open a Jira ticket and describe also the symptoms? This
> >> might be related, or the same issue: SPARK-11280 and also SPARK-11327
> >
> >
> > SPARK-11327 is exactly my problem, but I don't run docker.
> >
> > - Jo
> >
> >> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <
> jo.voordeckers@gmail.com>
> >> wrote:
> >>>
> >>>
> >>> Hi all,
> >>>
> >>> I'm running the mesos cluster dispatcher, however when I submit jobs
> with
> >>> things like jvm args, classpath order and UI port aren't added to the
> >>> commandline executed by the mesos scheduler. In fact it only cares
> about the
> >>> class, jar and num cores/mem.
> >>>
> >>>
> >>>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
> >>>
> >>> I've made an attempt at adding a few of the args that I believe are
> >>> useful to the MesosClusterScheduler class, which seems to solve my
> problem.
> >>>
> >>> Please have a look:
> >>>
> >>> https://github.com/apache/spark/pull/9752
> >>>
> >>> Thanks
> >>>
> >>> - Jo Voordeckers
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> --
> >> Iulian Dragos
> >>
> >> ------
> >> Reactive Apps on the JVM
> >> www.typesafe.com
> >>
> >
>
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Timothy Chen <tn...@gmail.com>.
Hi Jo,
Thanks for the links, I would expected the properties to be in
scheduler properties but I need to double check.
I'll be looking into these problems this week.
Tim
On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers
<jo...@gmail.com> wrote:
> On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <iu...@typesafe.com>
> wrote:
>>
>> I think it actually tries to send all properties as part of
>> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
>>
>>
>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
>>
>
> Aha that's interesting, I overlooked that line, I'll debug some more today
> because I know for sure that those options don't make it onto the
> commandline when I was running it in my debugger.
>
>>
>> Can you please open a Jira ticket and describe also the symptoms? This
>> might be related, or the same issue: SPARK-11280 and also SPARK-11327
>
>
> SPARK-11327 is exactly my problem, but I don't run docker.
>
> - Jo
>
>> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
>> wrote:
>>>
>>>
>>> Hi all,
>>>
>>> I'm running the mesos cluster dispatcher, however when I submit jobs with
>>> things like jvm args, classpath order and UI port aren't added to the
>>> commandline executed by the mesos scheduler. In fact it only cares about the
>>> class, jar and num cores/mem.
>>>
>>>
>>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>>>
>>> I've made an attempt at adding a few of the args that I believe are
>>> useful to the MesosClusterScheduler class, which seems to solve my problem.
>>>
>>> Please have a look:
>>>
>>> https://github.com/apache/spark/pull/9752
>>>
>>> Thanks
>>>
>>> - Jo Voordeckers
>>>
>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> ------
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Timothy Chen <tn...@gmail.com>.
Hi Jo,
Thanks for the links, I would expected the properties to be in
scheduler properties but I need to double check.
I'll be looking into these problems this week.
Tim
On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers
<jo...@gmail.com> wrote:
> On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <iu...@typesafe.com>
> wrote:
>>
>> I think it actually tries to send all properties as part of
>> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
>>
>>
>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
>>
>
> Aha that's interesting, I overlooked that line, I'll debug some more today
> because I know for sure that those options don't make it onto the
> commandline when I was running it in my debugger.
>
>>
>> Can you please open a Jira ticket and describe also the symptoms? This
>> might be related, or the same issue: SPARK-11280 and also SPARK-11327
>
>
> SPARK-11327 is exactly my problem, but I don't run docker.
>
> - Jo
>
>> On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
>> wrote:
>>>
>>>
>>> Hi all,
>>>
>>> I'm running the mesos cluster dispatcher, however when I submit jobs with
>>> things like jvm args, classpath order and UI port aren't added to the
>>> commandline executed by the mesos scheduler. In fact it only cares about the
>>> class, jar and num cores/mem.
>>>
>>>
>>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>>>
>>> I've made an attempt at adding a few of the args that I believe are
>>> useful to the MesosClusterScheduler class, which seems to solve my problem.
>>>
>>> Please have a look:
>>>
>>> https://github.com/apache/spark/pull/9752
>>>
>>> Thanks
>>>
>>> - Jo Voordeckers
>>>
>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> ------
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Jo Voordeckers <jo...@gmail.com>.
On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <iu...@typesafe.com>
wrote:
> I think it actually tries to send all properties as part of
> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
>
>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
>
>
Aha that's interesting, I overlooked that line, I'll debug some more today
because I know for sure that those options don't make it onto the
commandline when I was running it in my debugger.
> Can you please open a Jira ticket and describe also the symptoms? This
> might be related, or the same issue: SPARK-11280
> <https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327
> <https://issues.apache.org/jira/browse/SPARK-11327>
>
SPARK-11327 <https://issues.apache.org/jira/browse/SPARK-11327> is exactly
my problem, but I don't run docker.
- Jo
On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
> wrote:
>
>>
>> Hi all,
>>
>> I'm running the mesos cluster dispatcher, however when I submit jobs with
>> things like jvm args, classpath order and UI port aren't added to the
>> commandline executed by the mesos scheduler. In fact it only cares about
>> the class, jar and num cores/mem.
>>
>>
>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>>
>> I've made an attempt at adding a few of the args that I believe are
>> useful to the MesosClusterScheduler class, which seems to solve my problem.
>>
>> Please have a look:
>>
>> https://github.com/apache/spark/pull/9752
>>
>> Thanks
>>
>> - Jo Voordeckers
>>
>>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
>
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Jo Voordeckers <jo...@gmail.com>.
On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș <iu...@typesafe.com>
wrote:
> I think it actually tries to send all properties as part of
> `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
>
>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
>
>
Aha that's interesting, I overlooked that line, I'll debug some more today
because I know for sure that those options don't make it onto the
commandline when I was running it in my debugger.
> Can you please open a Jira ticket and describe also the symptoms? This
> might be related, or the same issue: SPARK-11280
> <https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327
> <https://issues.apache.org/jira/browse/SPARK-11327>
>
SPARK-11327 <https://issues.apache.org/jira/browse/SPARK-11327> is exactly
my problem, but I don't run docker.
- Jo
On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
> wrote:
>
>>
>> Hi all,
>>
>> I'm running the mesos cluster dispatcher, however when I submit jobs with
>> things like jvm args, classpath order and UI port aren't added to the
>> commandline executed by the mesos scheduler. In fact it only cares about
>> the class, jar and num cores/mem.
>>
>>
>> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>>
>> I've made an attempt at adding a few of the args that I believe are
>> useful to the MesosClusterScheduler class, which seems to solve my problem.
>>
>> Please have a look:
>>
>> https://github.com/apache/spark/pull/9752
>>
>> Thanks
>>
>> - Jo Voordeckers
>>
>>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
>
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Iulian Dragoș <iu...@typesafe.com>.
Hi Jo,
I agree that there's something fishy with the cluster dispatcher, I've seen
some issues like that.
I think it actually tries to send all properties as part of
`SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
Can you please open a Jira ticket and describe also the symptoms? This
might be related, or the same issue: SPARK-11280
<https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327
<https://issues.apache.org/jira/browse/SPARK-11327>
thanks,
iulian
On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
wrote:
>
> Hi all,
>
> I'm running the mesos cluster dispatcher, however when I submit jobs with
> things like jvm args, classpath order and UI port aren't added to the
> commandline executed by the mesos scheduler. In fact it only cares about
> the class, jar and num cores/mem.
>
>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>
> I've made an attempt at adding a few of the args that I believe are useful
> to the MesosClusterScheduler class, which seems to solve my problem.
>
> Please have a look:
>
> https://github.com/apache/spark/pull/9752
>
> Thanks
>
> - Jo Voordeckers
>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Mesos cluster dispatcher doesn't respect most args from the
submit req
Posted by Iulian Dragoș <iu...@typesafe.com>.
Hi Jo,
I agree that there's something fishy with the cluster dispatcher, I've seen
some issues like that.
I think it actually tries to send all properties as part of
`SPARK_EXECUTOR_OPTS`, which may not be everything that's needed:
https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L375-L377
Can you please open a Jira ticket and describe also the symptoms? This
might be related, or the same issue: SPARK-11280
<https://issues.apache.org/jira/browse/SPARK-11280> and also SPARK-11327
<https://issues.apache.org/jira/browse/SPARK-11327>
thanks,
iulian
On Tue, Nov 17, 2015 at 2:46 AM, Jo Voordeckers <jo...@gmail.com>
wrote:
>
> Hi all,
>
> I'm running the mesos cluster dispatcher, however when I submit jobs with
> things like jvm args, classpath order and UI port aren't added to the
> commandline executed by the mesos scheduler. In fact it only cares about
> the class, jar and num cores/mem.
>
>
> https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L412-L424
>
> I've made an attempt at adding a few of the args that I believe are useful
> to the MesosClusterScheduler class, which seems to solve my problem.
>
> Please have a look:
>
> https://github.com/apache/spark/pull/9752
>
> Thanks
>
> - Jo Voordeckers
>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com