clientMode in RpcEnv.create in Spark on YARN vs general case (driver vs executors)?

[Jacek Laskowski <ja...@japila.pl>]

Hi,

I'm trying to get the gist of clientMode input parameter for
RpcEnv.create [1]. It is disabled (i.e. false) by default.

I've managed to find out that, in the "general" case, it's enabled for
executors and disabled for the driver.

(it's also used for Spark Standalone's master and workers but it's
infra and I'm not interested in exploring it currently).

I've however noticed that in YARN the clientMode parameter means
something more, i.e. whether the Spark application runs in client or
cluster deploy mode.

In YARN my understanding of the parameter is that clientMode is
enabled explicitly when Spark on YARN's ApplicationMaster creates the
`sparkYarnAM` RPC Environment [2] (when executed for client deploy
mode [3])

This is because in client deploy mode the driver runs on some other
node and the AM acts simply as a proxy to Spark executors that run in
their own YARN containers. This is (also?) because in client deploy
mode in Spark on YARN we have separate JVM processes for the driver,
the AM and Spark executors. The distinction is

Is the last two paragraphs correct?

I'd appreciate if you could fix and fill out the gaps where necessary.
Thanks a lot to make it so much easier for me and...participants of my
Spark workshops ;-)

[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rpc/RpcEnv.scala#L42

[2] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L434

[3] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L254

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org

Re: clientMode in RpcEnv.create in Spark on YARN vs general case (driver vs executors)?

[Marcelo Vanzin <va...@cloudera.com>]

On Wed, Jan 18, 2017 at 1:29 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> I'm trying to get the gist of clientMode input parameter for
> RpcEnv.create [1]. It is disabled (i.e. false) by default.

"clientMode" means whether the RpcEnv only opens external connections
(client) or also accepts incoming connections. From that you should be
able to understand why it's true or false in the cases you looked at.


-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org