[YARN] $ and $$ in prepareCommand to resolve environment in ExecutorRunnable?

[Jacek Laskowski <ja...@japila.pl>]

Hi,

Just noticed that [1] (and also [3]) is very cautious with $ and $$ to
expand environment variables.

    javaOpts += "-Djava.io.tmpdir=" +
      new Path(
        YarnSparkHadoopUtil.expandEnvironment(Environment.PWD), // <-- here
        YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR
      )

Few lines below in the same method [2] the code doesn't seem to bother
to call $ directly :

Client.buildPath(Environment.PWD.$(), uri.getPath())

Why? Any particular reason for the difference in the two lines? #curious

[1] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L156

[2] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L203

[3] https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L210

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org