You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Jahagirdar, Madhu" <ma...@philips.com> on 2015/07/20 04:52:11 UTC
Spark Mesos Dispatcher
All,
Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ?
Regards,
Madhu Jahagirdar
________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: Spark Mesos Dispatcher
Posted by Tim Chen <ti...@mesosphere.io>.
Depends on how you run 1.3/1.4 versions of Spark, if you're giving it
different Docker images / tar balls of Spark, technically it should work
since it's just launching a driver for you at the end of the day.
However, I haven't really tried it so let me know if you run into problems
with it.
Tim
On Sun, Jul 19, 2015 at 9:23 PM, Jerry Lam <ch...@gmail.com> wrote:
> I only used client mode both 1.3 and 1.4 versions on mesos.
> I skimmed through
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala.
> I would actually backport the Cluster Mode feature. Sorry, I don't have an
> answer for this.
>
>
> On Sun, Jul 19, 2015 at 11:16 PM, Jahagirdar, Madhu <
> madhu.jahagirdar@philips.com> wrote:
>
>> 1.3 does not have MesosDisptacher or does not have support for Mesos
>> cluster mode , is it still possible to create a Dispatcher using 1.4 and
>> run 1.3 using that dispatcher ?
>> ------------------------------
>> *From:* Jerry Lam [chilinglam@gmail.com]
>> *Sent:* Monday, July 20, 2015 8:27 AM
>> *To:* Jahagirdar, Madhu
>> *Cc:* user; dev@spark.apache.org
>> *Subject:* Re: Spark Mesos Dispatcher
>>
>> Yes.
>>
>> Sent from my iPhone
>>
>> On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <
>> madhu.jahagirdar@philips.com> wrote:
>>
>> All,
>>
>> Can we run different version of Spark using the same Mesos Dispatcher.
>> For example we can run drivers with Spark 1.3 and Spark 1.4 at the same
>> time ?
>>
>> Regards,
>> Madhu Jahagirdar
>>
>> ------------------------------
>> The information contained in this message may be confidential and legally
>> protected under applicable law. The message is intended solely for the
>> addressee(s). If you are not the intended recipient, you are hereby
>> notified that any use, forwarding, dissemination, or reproduction of this
>> message is strictly prohibited and may be unlawful. If you are not the
>> intended recipient, please contact the sender by return e-mail and destroy
>> all copies of the original message.
>>
>>
>
Re: Spark Mesos Dispatcher
Posted by Jerry Lam <ch...@gmail.com>.
I only used client mode both 1.3 and 1.4 versions on mesos.
I skimmed through
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala.
I would actually backport the Cluster Mode feature. Sorry, I don't have an
answer for this.
On Sun, Jul 19, 2015 at 11:16 PM, Jahagirdar, Madhu <
madhu.jahagirdar@philips.com> wrote:
> 1.3 does not have MesosDisptacher or does not have support for Mesos
> cluster mode , is it still possible to create a Dispatcher using 1.4 and
> run 1.3 using that dispatcher ?
> ------------------------------
> *From:* Jerry Lam [chilinglam@gmail.com]
> *Sent:* Monday, July 20, 2015 8:27 AM
> *To:* Jahagirdar, Madhu
> *Cc:* user; dev@spark.apache.org
> *Subject:* Re: Spark Mesos Dispatcher
>
> Yes.
>
> Sent from my iPhone
>
> On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <
> madhu.jahagirdar@philips.com> wrote:
>
> All,
>
> Can we run different version of Spark using the same Mesos Dispatcher.
> For example we can run drivers with Spark 1.3 and Spark 1.4 at the same
> time ?
>
> Regards,
> Madhu Jahagirdar
>
> ------------------------------
> The information contained in this message may be confidential and legally
> protected under applicable law. The message is intended solely for the
> addressee(s). If you are not the intended recipient, you are hereby
> notified that any use, forwarding, dissemination, or reproduction of this
> message is strictly prohibited and may be unlawful. If you are not the
> intended recipient, please contact the sender by return e-mail and destroy
> all copies of the original message.
>
>
Re: Spark Mesos Dispatcher
Posted by Jerry Lam <ch...@gmail.com>.
I only used client mode both 1.3 and 1.4 versions on mesos.
I skimmed through
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcher.scala.
I would actually backport the Cluster Mode feature. Sorry, I don't have an
answer for this.
On Sun, Jul 19, 2015 at 11:16 PM, Jahagirdar, Madhu <
madhu.jahagirdar@philips.com> wrote:
> 1.3 does not have MesosDisptacher or does not have support for Mesos
> cluster mode , is it still possible to create a Dispatcher using 1.4 and
> run 1.3 using that dispatcher ?
> ------------------------------
> *From:* Jerry Lam [chilinglam@gmail.com]
> *Sent:* Monday, July 20, 2015 8:27 AM
> *To:* Jahagirdar, Madhu
> *Cc:* user; dev@spark.apache.org
> *Subject:* Re: Spark Mesos Dispatcher
>
> Yes.
>
> Sent from my iPhone
>
> On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <
> madhu.jahagirdar@philips.com> wrote:
>
> All,
>
> Can we run different version of Spark using the same Mesos Dispatcher.
> For example we can run drivers with Spark 1.3 and Spark 1.4 at the same
> time ?
>
> Regards,
> Madhu Jahagirdar
>
> ------------------------------
> The information contained in this message may be confidential and legally
> protected under applicable law. The message is intended solely for the
> addressee(s). If you are not the intended recipient, you are hereby
> notified that any use, forwarding, dissemination, or reproduction of this
> message is strictly prohibited and may be unlawful. If you are not the
> intended recipient, please contact the sender by return e-mail and destroy
> all copies of the original message.
>
>
RE: Spark Mesos Dispatcher
Posted by "Jahagirdar, Madhu" <ma...@philips.com>.
1.3 does not have MesosDisptacher or does not have support for Mesos cluster mode , is it still possible to create a Dispatcher using 1.4 and run 1.3 using that dispatcher ?
________________________________
From: Jerry Lam [chilinglam@gmail.com]
Sent: Monday, July 20, 2015 8:27 AM
To: Jahagirdar, Madhu
Cc: user; dev@spark.apache.org
Subject: Re: Spark Mesos Dispatcher
Yes.
Sent from my iPhone
On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <ma...@philips.com>> wrote:
All,
Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ?
Regards,
Madhu Jahagirdar
________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: Spark Mesos Dispatcher
Posted by Jerry Lam <ch...@gmail.com>.
Yes.
Sent from my iPhone
> On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <ma...@philips.com> wrote:
>
> All,
>
> Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ?
>
> Regards,
> Madhu Jahagirdar
>
> The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
Re: Spark Mesos Dispatcher
Posted by Jerry Lam <ch...@gmail.com>.
Yes.
Sent from my iPhone
> On 19 Jul, 2015, at 10:52 pm, "Jahagirdar, Madhu" <ma...@philips.com> wrote:
>
> All,
>
> Can we run different version of Spark using the same Mesos Dispatcher. For example we can run drivers with Spark 1.3 and Spark 1.4 at the same time ?
>
> Regards,
> Madhu Jahagirdar
>
> The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.