You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nisrina Luthfiyati <ni...@gmail.com> on 2015/12/07 16:01:38 UTC
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
TaskSetManager and it does looked like tasks are directly sent to
executors. Also would love to be corrected if mistaken as I have little
knowledge about Spark internals and very new at scala.
On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
> nisrina.luthfiyati@gmail.com> wrote:
>
>> Hi all,
>> I'm trying to understand how yarn-client mode works and found these two
>> diagrams:
>>
>>
>>
>>
>> In the first diagram, it looks like the driver running in client directly
>> communicates with executors to issue application commands, while in the
>> second diagram it looks like application commands is sent to application
>> master first and then forwarded to executors.
>>
>
> My limited understanding tells me that regardless of deploy mode (local,
> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
> TaskSets to executors once they're launched. YARN and Mesos are only used
> until they offer resources (CPU and memory) and once executors start, these
> cluster managers are not engaged in the communication (driver and executors
> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>
> I'd love being corrected if mistaken. Thanks.
>
> Jacek
>
--
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina
Re: In yarn-client mode, is it the driver or application master that issue commands to executors?
Posted by Ali Tajeldin EDU <al...@gmail.com>.
Checkout the Sameer Farooqui video on youtube for spark internals (https://www.youtube.com/watch?v=7ooZ4S7Ay6Y&list=PLIxzgeMkSrQ-2Uizm4l0HjNSSy2NxgqjX)
Starting at 2:15:00, he describes YARN mode.
btw, highly recommend the entire video. Very detailed and concise.
--
Ali
On Dec 7, 2015, at 8:38 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> That's my understanding, too. Just spent an entire morning today to check it out and would be surprised to hear otherwise.
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | https://medium.com/@jaceklaskowski/ | http://blog.jaceklaskowski.pl
> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
> On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati <ni...@gmail.com> wrote:
> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and TaskSetManager and it does looked like tasks are directly sent to executors. Also would love to be corrected if mistaken as I have little knowledge about Spark internals and very new at scala.
>
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <ni...@gmail.com> wrote:
> Hi all,
> I'm trying to understand how yarn-client mode works and found these two diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly communicates with executors to issue application commands, while in the second diagram it looks like application commands is sent to application master first and then forwarded to executors.
>
> My limited understanding tells me that regardless of deploy mode (local, standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends TaskSets to executors once they're launched. YARN and Mesos are only used until they offer resources (CPU and memory) and once executors start, these cluster managers are not engaged in the communication (driver and executors communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>
> I'd love being corrected if mistaken. Thanks.
>
> Jacek
>
>
>
> --
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
>
>
Re: In yarn-client mode, is it the driver or application master that
issue commands to executors?
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
That's my understanding, too. Just spent an entire morning today to check
it out and would be surprised to hear otherwise.
Pozdrawiam,
Jacek
--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati <
nisrina.luthfiyati@gmail.com> wrote:
> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
> TaskSetManager and it does looked like tasks are directly sent to
> executors. Also would love to be corrected if mistaken as I have little
> knowledge about Spark internals and very new at scala.
>
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
>> nisrina.luthfiyati@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm trying to understand how yarn-client mode works and found these two
>>> diagrams:
>>>
>>>
>>>
>>>
>>> In the first diagram, it looks like the driver running in client
>>> directly communicates with executors to issue application commands, while
>>> in the second diagram it looks like application commands is sent to
>>> application master first and then forwarded to executors.
>>>
>>
>> My limited understanding tells me that regardless of deploy mode (local,
>> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
>> TaskSets to executors once they're launched. YARN and Mesos are only used
>> until they offer resources (CPU and memory) and once executors start, these
>> cluster managers are not engaged in the communication (driver and executors
>> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>>
>> I'd love being corrected if mistaken. Thanks.
>>
>> Jacek
>>
>
>
>
> --
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
>
>