You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/28 11:16:00 UTC

[GitHub] [spark] ScrapCodes opened a new pull request #27735: [SPARK-30985][SPARK-25065][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.

ScrapCodes opened a new pull request #27735: [SPARK-30985][SPARK-25065][k8s] Support propagating SPARK_CONF_DIR files to driver and executor pods.
URL: https://github.com/apache/spark/pull/27735

### What changes were proposed in this pull request?
This is an improvement, we mount all the files(except the templates and spark properties files) from `SPARK_CONF_DIR` to executor and driver pods.

### Why are the changes needed?
[verbatim paste from the JIRA.]
`SPARK_CONF_DIR` hosts configuration files like,
1) `spark-defaults.conf` - containing all the spark properties.
2) `log4j.properties` - Logger configuration.
3) `spark-env.sh` - Environment variables to be setup at driver and executor.
4) `core-site.xml` - Hadoop related configuration.
5) `fairscheduler.xml` - Spark's fair scheduling policy at the job level.
6) `metrics.properties` - Spark metrics.
7) Any user specific - library or framework specific configuration file.

Traditionally, `SPARK_CONF_DIR` has been the home to all user specific configuration files and the default behaviour in the Yarn or standalone mode is that these configuration files are copied to the worker nodes as required by the users themselves. In other words, they are not auto-copied.
But, in the case of spark on kubernetes, we use spark images and generally these images are approved or undergoe some kind of standardisation. These files cannot be simply copied to the SPARK_CONF_DIR of the running executor and driver pods by the user.

So, at the moment we have special casing for providing each configuration and for any other user specific configuration files, the process is more complex, i.e. - e.g. one can start with their own custom image of spark with configuration files pre installed etc..
Examples of special casing are:
1. Hadoop configuration in spark.kubernetes.hadoop.configMapName
2. Spark-env.sh as in spark.kubernetes.driverEnv.[EnvironmentVariableName]
3. Log4j.properties as in https://github.com/apache/spark/pull/26193
... And for those such special casing does not exist, they are simply out of luck.

So this feature, will let the user specific configuration files be mounted on the driver and executor pods' `SPARK_CONF_DIR`.

### Further scope
At the moment it is not clear, if there is a need to, let user specify which config files to propagate - to driver and or executor. But, if there is a case that feature will be helpful, we can increase the scope of this work or create another JIRA issue to track that work.

### Does this PR introduce any user-facing change?
Yes, previously these configuration files were not propagated by default, now after this patch it is propagated. There is also a configuration `spark.kubernetes.propagateConf` to turn off this feature and return to default behaviour.

### How was this patch tested?
Added tests.

Also manually tested, by deploying it to a minikube cluster and observing the additional configs were mounted in the logs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org