You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nisrina Luthfiyati <ni...@gmail.com> on 2015/11/27 12:12:02 UTC
In yarn-client mode, is it the driver or application master that
issue commands to executors?
Hi all,
I'm trying to understand how yarn-client mode works and found these two
diagrams:
In the first diagram, it looks like the driver running in client directly
communicates with executors to issue application commands, while in the
second diagram it looks like application commands is sent to application
master first and then forwarded to executors.
Would anyone knows which case is true or is there any other interpretation
to these diagrams?
Thanks!
Nisrina
Re: In yarn-client mode, is it the driver or application master that issue commands to executors?
Posted by Ali Tajeldin EDU <al...@gmail.com>.
Checkout the Sameer Farooqui video on youtube for spark internals (https://www.youtube.com/watch?v=7ooZ4S7Ay6Y&list=PLIxzgeMkSrQ-2Uizm4l0HjNSSy2NxgqjX)
Starting at 2:15:00, he describes YARN mode.
btw, highly recommend the entire video. Very detailed and concise.
--
Ali
On Dec 7, 2015, at 8:38 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> That's my understanding, too. Just spent an entire morning today to check it out and would be surprised to hear otherwise.
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | https://medium.com/@jaceklaskowski/ | http://blog.jaceklaskowski.pl
> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
> On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati <ni...@gmail.com> wrote:
> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and TaskSetManager and it does looked like tasks are directly sent to executors. Also would love to be corrected if mistaken as I have little knowledge about Spark internals and very new at scala.
>
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <ni...@gmail.com> wrote:
> Hi all,
> I'm trying to understand how yarn-client mode works and found these two diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly communicates with executors to issue application commands, while in the second diagram it looks like application commands is sent to application master first and then forwarded to executors.
>
> My limited understanding tells me that regardless of deploy mode (local, standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends TaskSets to executors once they're launched. YARN and Mesos are only used until they offer resources (CPU and memory) and once executors start, these cluster managers are not engaged in the communication (driver and executors communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>
> I'd love being corrected if mistaken. Thanks.
>
> Jacek
>
>
>
> --
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
>
>
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
That's my understanding, too. Just spent an entire morning today to check
it out and would be surprised to hear otherwise.
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati <
nisrina.luthfiyati@gmail.com> wrote:
> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
> TaskSetManager and it does looked like tasks are directly sent to
> executors. Also would love to be corrected if mistaken as I have little
> knowledge about Spark internals and very new at scala.
>
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
>> nisrina.luthfiyati@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm trying to understand how yarn-client mode works and found these two
>>> diagrams:
>>>
>>>
>>>
>>>
>>> In the first diagram, it looks like the driver running in client
>>> directly communicates with executors to issue application commands, while
>>> in the second diagram it looks like application commands is sent to
>>> application master first and then forwarded to executors.
>>>
>>
>> My limited understanding tells me that regardless of deploy mode (local,
>> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
>> TaskSets to executors once they're launched. YARN and Mesos are only used
>> until they offer resources (CPU and memory) and once executors start, these
>> cluster managers are not engaged in the communication (driver and executors
>> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>>
>> I'd love being corrected if mistaken. Thanks.
>>
>> Jacek
>>
>
>
>
> --
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
>
>
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Posted by Nisrina Luthfiyati <ni...@gmail.com>.
Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
TaskSetManager and it does looked like tasks are directly sent to
executors. Also would love to be corrected if mistaken as I have little
knowledge about Spark internals and very new at scala.
On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
> nisrina.luthfiyati@gmail.com> wrote:
>
>> Hi all,
>> I'm trying to understand how yarn-client mode works and found these two
>> diagrams:
>>
>>
>>
>>
>> In the first diagram, it looks like the driver running in client directly
>> communicates with executors to issue application commands, while in the
>> second diagram it looks like application commands is sent to application
>> master first and then forwarded to executors.
>>
>
> My limited understanding tells me that regardless of deploy mode (local,
> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
> TaskSets to executors once they're launched. YARN and Mesos are only used
> until they offer resources (CPU and memory) and once executors start, these
> cluster managers are not engaged in the communication (driver and executors
> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>
> I'd love being corrected if mistaken. Thanks.
>
> Jacek
>
--
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Posted by Jacek Laskowski <ja...@japila.pl>.
On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
nisrina.luthfiyati@gmail.com> wrote:
> Hi all,
> I'm trying to understand how yarn-client mode works and found these two
> diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly
> communicates with executors to issue application commands, while in the
> second diagram it looks like application commands is sent to application
> master first and then forwarded to executors.
>
My limited understanding tells me that regardless of deploy mode (local,
standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
TaskSets to executors once they're launched. YARN and Mesos are only used
until they offer resources (CPU and memory) and once executors start, these
cluster managers are not engaged in the communication (driver and executors
communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
I'd love being corrected if mistaken. Thanks.
Jacek
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Posted by Nisrina Luthfiyati <ni...@gmail.com>.
Hi Mich, thank you for the answer. Regarding the diagrams, I'm specifically
referring to the direct line between spark yarn client to spark executor in
the first diagram which implies direct communication to executor when
issuing application commands. And the 'Application commands' & 'Issue
application commands' lines in the second diagram which implies that spark
driver in client communicates to executor via yarn application master
(Correct me if i'm wrong in these interpretations). Would you happen to
know how spark drivers communicates with executor in yarn-client mode or if
both can be true under different circumstances?
Thanks again,
Nisrina.
On Fri, Nov 27, 2015 at 6:22 PM, Mich Talebzadeh <mi...@peridale.co.uk>
wrote:
> Hi,
>
>
>
> In general YARN is used as the resource scheduler regardless of the
> execution engine whether it is MapReduce or Spark.
>
>
>
> Yarn will create a resource container for the submitted job (that is the
> Spark client) and will execute it in the default engine (in this case
> Spark). There will be a job scheduler and one or more Spark Executors
> depending on the cluster. So as far as I can see both diagrams are correct,
>
>
>
> HTH
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Nisrina Luthfiyati [mailto:nisrina.luthfiyati@gmail.com]
> *Sent:* 27 November 2015 11:12
> *To:* user@spark.apache.org
> *Subject:* In yarn-client mode, is it the driver or application master
> that issue commands to executors?
>
>
>
> Hi all,
>
> I'm trying to understand how yarn-client mode works and found these two
> diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly
> communicates with executors to issue application commands, while in the
> second diagram it looks like application commands is sent to application
> master first and then forwarded to executors.
>
> Would anyone knows which case is true or is there any other interpretation
> to these diagrams?
>
> Thanks!
>
> Nisrina
>
--
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina
RE: In yarn-client mode, is it the driver or application master that issue commands to executors?
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi,
In general YARN is used as the resource scheduler regardless of the execution engine whether it is MapReduce or Spark.
Yarn will create a resource container for the submitted job (that is the Spark client) and will execute it in the default engine (in this case Spark). There will be a job scheduler and one or more Spark Executors depending on the cluster. So as far as I can see both diagrams are correct,
HTH
Mich Talebzadeh
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
<http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly
<http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Nisrina Luthfiyati [mailto:nisrina.luthfiyati@gmail.com]
Sent: 27 November 2015 11:12
To: user@spark.apache.org
Subject: In yarn-client mode, is it the driver or application master that issue commands to executors?
Hi all,
I'm trying to understand how yarn-client mode works and found these two diagrams:
<http://image.slidesharecdn.com/th-1150a-hall1-feng-v2-140617142634-phpapp01/95/sparkonyarn-empower-spark-applications-on-hadoop-cluster-10-638.jpg?cb=1403015417>
<http://blog.cloudera.com/wp-content/uploads/2014/05/spark-yarn-f22.png>
In the first diagram, it looks like the driver running in client directly communicates with executors to issue application commands, while in the second diagram it looks like application commands is sent to application master first and then forwarded to executors.
Would anyone knows which case is true or is there any other interpretation to these diagrams?
Thanks!
Nisrina