You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Bill Sparks <js...@cray.com> on 2013/11/14 17:56:27 UTC
SPARK + YARN the general case
Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
Thanks,
Bill
RE: SPARK + YARN the general case
Posted by "Liu, Raymond" <ra...@intel.com>.
Well, with #101 pull request get merged. Shark on yarn and spark streaming on Yarn should both working.
I do verify both these working modes with simple test case when I submit the #101 request, while codes changes a lot after that. Might have a few little things to fix.
Best Regards,
Raymond Liu
From: Tom Graves [mailto:tgraves_cs@yahoo.com]
Sent: Saturday, November 16, 2013 5:57 AM
To: user@spark.incubator.apache.org
Subject: Re: SPARK + YARN the general case
Yes that is correct. It has a static set of nodes currently. We want to make that more dynamic in the future also.
Tom
On Friday, November 15, 2013 2:16 PM, Michael (Bach) Bui <fr...@adatao.com>> wrote:
Tom, more on Shark type of applications on Yarn.
In the current implementation, during the duration of a SparkContext execution, Yarn will give an unchanged set of nodes to the SparkContext, is that right?
If that is the case, IMO, it may not be the best architecture for Shark, because users may load data from nodes that are not in the given set of nodes. Am I right?
On Nov 15, 2013, at 12:51 PM, Tom Graves <tg...@yahoo.com>> wrote:
Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet.
Tom
On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <fr...@adatao.com>> wrote:
Hi Tom,
I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right?
Thanks.
On Nov 15, 2013, at 8:15 AM, Tom Graves <tg...@yahoo.com>> wrote:
Hey Bill,
Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
There is a pull request (101) to get spark shell working with YARN.
Tom
On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com>> wrote:
Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
Thanks,
Bill
Re: SPARK + YARN the general case
Posted by Tom Graves <tg...@yahoo.com>.
Yes that is correct. It has a static set of nodes currently. We want to make that more dynamic in the future also.
Tom
On Friday, November 15, 2013 2:16 PM, Michael (Bach) Bui <fr...@adatao.com> wrote:
Tom, more on Shark type of applications on Yarn.
In the current implementation, during the duration of a SparkContext execution, Yarn will give an unchanged set of nodes to the SparkContext, is that right?
If that is the case, IMO, it may not be the best architecture for Shark, because users may load data from nodes that are not in the given set of nodes. Am I right?
On Nov 15, 2013, at 12:51 PM, Tom Graves <tg...@yahoo.com> wrote:
Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet.
>
>
>Tom
>
>
>
>On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <fr...@adatao.com> wrote:
>
>Hi Tom,
>
>
>I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right?
>Thanks.
>
>
>
>On Nov 15, 2013, at 8:15 AM, Tom Graves <tg...@yahoo.com> wrote:
>
>Hey Bill,
>>
>>
>>Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
>>
>>
>>There is a pull request (101) to get spark shell working with YARN.
>>
>>
>>Tom
>>
>>
>>
>>On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
>>
>>Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
>>
>>
>>Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
>>
>>
>>Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
>>
>>
>>Thanks,
>> Bill
>>
>>
>
>
>
Re: SPARK + YARN the general case
Posted by "Michael (Bach) Bui" <fr...@adatao.com>.
Tom, more on Shark type of applications on Yarn.
In the current implementation, during the duration of a SparkContext execution, Yarn will give an unchanged set of nodes to the SparkContext, is that right?
If that is the case, IMO, it may not be the best architecture for Shark, because users may load data from nodes that are not in the given set of nodes. Am I right?
On Nov 15, 2013, at 12:51 PM, Tom Graves <tg...@yahoo.com> wrote:
> Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet.
>
> Tom
>
>
> On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <fr...@adatao.com> wrote:
> Hi Tom,
>
> I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right?
> Thanks.
>
>
>
> On Nov 15, 2013, at 8:15 AM, Tom Graves <tg...@yahoo.com> wrote:
>
>> Hey Bill,
>>
>> Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
>>
>> There is a pull request (101) to get spark shell working with YARN.
>>
>> Tom
>>
>>
>> On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
>> Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
>>
>> Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
>>
>> Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
>>
>> Thanks,
>> Bill
>>
>>
>
>
>
Re: SPARK + YARN the general case
Posted by Tom Graves <tg...@yahoo.com>.
Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet.
Tom
On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <fr...@adatao.com> wrote:
Hi Tom,
I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right?
Thanks.
On Nov 15, 2013, at 8:15 AM, Tom Graves <tg...@yahoo.com> wrote:
Hey Bill,
>
>
>Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
>
>
>There is a pull request (101) to get spark shell working with YARN.
>
>
>Tom
>
>
>
>On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
>
>Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
>
>
>Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
>
>
>Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
>
>
>Thanks,
> Bill
>
>
Re: SPARK + YARN the general case
Posted by "Michael (Bach) Bui" <fr...@adatao.com>.
Hi Tom,
I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right?
Thanks.
On Nov 15, 2013, at 8:15 AM, Tom Graves <tg...@yahoo.com> wrote:
> Hey Bill,
>
> Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
>
> There is a pull request (101) to get spark shell working with YARN.
>
> Tom
>
>
> On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
> Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
>
> Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
>
> Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
>
> Thanks,
> Bill
>
>
Re: SPARK + YARN the general case
Posted by Tom Graves <tg...@yahoo.com>.
Hey Phillip,
I haven't actually tried spark streaming on YARN at this point so I can't say for sure, but as you say just from what I've read I don't see anything that would prevent it from working.
Since I don't know of anyone that has tried it I wouldn't be surprised it atleast something small needs to be fixed to support it though.
Tom
On Friday, November 15, 2013 9:40 AM, Philip Ogren <ph...@oracle.com> wrote:
Tom,
Can you just clarify that when you say "Spark on Yarn only supports
batch mode" that you are not excluding Spark Streaming from working
with Yarn? A quick scan of the Spark Streaming documentation makes
no mention of Yarn, but I thought that this should be possible.
Thanks,
Philip
On 11/15/2013 7:15 AM, Tom Graves wrote:
Hey Bill,
>
>
>Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
>
>
>There is a pull request (101) to get spark shell working with YARN.
>
>
>Tom
>
>
>
>On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
>
>Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
>
>
>Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
>
>
>Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
>
>
>Thanks,
> Bill
>
>
Re: SPARK + YARN the general case
Posted by Philip Ogren <ph...@oracle.com>.
Tom,
Can you just clarify that when you say "Spark on Yarn only supports
batch mode" that you are not excluding Spark Streaming from working with
Yarn? A quick scan of the Spark Streaming documentation makes no
mention of Yarn, but I thought that this should be possible.
Thanks,
Philip
On 11/15/2013 7:15 AM, Tom Graves wrote:
> Hey Bill,
>
> Currently the Spark on Yarn only supports batch mode where you submit
> your job via the yarn Client. Note that this will hook the spark UI
> up to the Yarn ResourceManager web UI. Is there something more you
> were looking for then just finding the spark web ui for various jobs?
>
> There is a pull request (101) to get spark shell working with YARN.
>
> Tom
>
>
> On Thursday, November 14, 2013 10:57 AM, Bill Sparks
> <js...@cray.com> wrote:
> Sorry for the following question, but I just need a little clarity on
> expectations of Spark using YARN.
>
> Is it possible to use the spark-shell with YARN ? Or is the only way
> to submit a Spark job to YARN is by write a Java application and
> submit it via the yarn.Client application.
>
> Also is there a way of running the Spark master so that it can
> communicate with YARN so I can use the web UI for job tracking.
>
> Thanks,
> Bill
>
>
Re: SPARK + YARN the general case
Posted by Bill Sparks <js...@cray.com>.
Thanks for the reply.
--
Jonathan (Bill) Sparks
Software Architecture
Cray
From: Tom Graves <tg...@yahoo.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>, Tom Graves <tg...@yahoo.com>>
Date: Friday, November 15, 2013 8:15 AM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: Re: SPARK + YARN the general case
Hey Bill,
Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
There is a pull request (101) to get spark shell working with YARN.
Tom
On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com>> wrote:
Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
Thanks,
Bill
Re: SPARK + YARN the general case
Posted by Tom Graves <tg...@yahoo.com>.
Hey Bill,
Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs?
There is a pull request (101) to get spark shell working with YARN.
Tom
On Thursday, November 14, 2013 10:57 AM, Bill Sparks <js...@cray.com> wrote:
Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN.
Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application.
Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking.
Thanks,
Bill