You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/20 10:42:21 UTC
[GitHub] [spark] zhengchenyu opened a new pull request, #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
zhengchenyu opened a new pull request, #37949:
URL: https://github.com/apache/spark/pull/37949
https://issues.apache.org/jira/browse/SPARK-40504
### What changes were proposed in this pull request?
After apply this, AppMaster will load __spark_hadoop_conf__.xml to override the config. It means appmaster will use config from client.
### How was this patch tested?
manual test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
mridulm commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1309308465
+CC @xkrogen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1309412598
I'm a bit confused about why this change is necessary. `yarn.Client` already gathers all Hadoop config files under `HADOOP_CONF_DIR` (and `SPARK_CONF_DIR`) and uploads them, placing them on the classpath of all YARN containers:
https://github.com/apache/spark/blob/5600bef0ee6149ebc1abcf4c9c9b2991553ca3de/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L823-L842
So when the `new Configuration` object is created within `SparkHadoopUtil.newConfiguration`, it will already have access to the `yarn-site.xml` from your client side. That should override whatever configs are coming from the cluster side, since Spark puts Hadoop at the end of the classpath for containers.
Unless your `yarn-site.xml` is in `YARN_CONF_DIR` instead of `HADOOP_CONF_DIR`, I guess?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1311193343
@xkrogen Thanks for your review. In our cluster, YARN_CONF_DIR is same with HADOOP_CONF_DIR.
I add some key information about the failed application.
```
# some key information in launch_container.sh:
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/ke/conf/devtest/server/yarn/hadoop"}
export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/home/ke/conf/devtest/client-devtest/bigdata/hadoop:/home/ke/bin/hadoop/share/hadoop/common/lib/*:/home/ke/bin/hadoop/share/hadoop/common/*:/home/ke/bin/hadoop/share/hadoop/hdfs:/home/ke/bin/hadoop/share/hadoop/hdfs/lib/*:/home/ke/bin/hadoop/share/hadoop/hdfs/*:/home/ke/bin/hadoop/share/hadoop/mapreduce/lib/*:/home/ke/bin/hadoop/share/hadoop/mapreduce/*:/home/ke/bin/hadoop/share/hadoop/yarn:/home/ke/bin/hadoop/share/hadoop/yarn/lib/*:/home/ke/bin/hadoop/share/hadoop/yarn/*:/home/ke/conf/devtest/client-devtest/bigdata/tez:/home/ke/bin/tez/hadoop-shim-0.9.2.jar:/home
/ke/bin/tez/hadoop-shim-2.7-0.9.2.jar:/home/ke/bin/tez/lib:/home/ke/bin/tez/LICENSE:/home/ke/bin/tez/LICENSE-BSD-3clause:/home/ke/bin/tez/LICENSE-CDDLv1.0:/home/ke/bin/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/home/ke/bin/tez/LICENSE-MIT:/home/ke/bin/tez/LICENSE-SIL_OpenFontLicense-v1.1:/home/ke/bin/tez/NOTICE:/home/ke/bin/tez/tez-api-0.9.2.jar:/home/ke/bin/tez/tez-build-tools-0.9.2.jar:/home/ke/bin/tez/tez-common-0.9.2.jar:/home/ke/bin/tez/tez-dag-0.9.2.jar:/home/ke/bin/tez/tez-examples-0.9.2.jar:/home/ke/bin/tez/tez-ext-service-tests-0.9.2.jar:/home/ke/bin/tez/tez-history-parser-0.9.2.jar:/home/ke/bin/tez/tez-javadoc-tools-0.9.2.jar:/home/ke/bin/tez/tez-job-analyzer-0.9.2.jar:/home/ke/bin/tez/tez-mapreduce-0.9.2.jar:/home/ke/bin/tez/tez-protobuf-history-plugin-0.9.2.jar:/home/ke/bin/tez/tez-runtime-internals-0.9.2.jar:/home/ke/bin/tez/tez-runtime-library-0.9.2.jar:/home/ke/bin/tez/tez-tests-0.9.2.jar:/home/ke/bin/tez/tez-ui-0.9.2.war:/home/ke/bin/tez/tez-yarn-timeline-history-0.9.2.jar:
/home/ke/bin/tez/tez-yarn-timeline-history-with-acls-0.9.2.jar:/home/ke/bin/tez/lib/accessors-smart-1.2.jar:/home/ke/bin/tez/lib/animal-sniffer-annotations-1.17.jar:/home/ke/bin/tez/lib/aopalliance-1.0.jar:/home/ke/bin/tez/lib/asm-5.0.4.jar:/home/ke/bin/tez/lib/async-http-client-1.8.16.jar:/home/ke/bin/tez/lib/audience-annotations-0.5.0.jar:/home/ke/bin/tez/lib/avro-1.7.7.jar:/home/ke/bin/tez/lib/bcpkix-jdk15on-1.60.jar:/home/ke/bin/tez/lib/bcprov-jdk15on-1.60.jar:/home/ke/bin/tez/lib/checker-qual-2.5.2.jar:/home/ke/bin/tez/lib/commons-beanutils-1.9.3.jar:/home/ke/bin/tez/lib/commons-cli-1.2.jar:/home/ke/bin/tez/lib/commons-codec-1.4.jar:/home/ke/bin/tez/lib/commons-collections-3.2.2.jar:/home/ke/bin/tez/lib/commons-collections4-4.1.jar:/home/ke/bin/tez/lib/commons-compress-1.18.jar:/home/ke/bin/tez/lib/commons-configuration2-2.1.1.jar:/home/ke/bin/tez/lib/commons-daemon-1.0.13.jar:/home/ke/bin/tez/lib/commons-io-2.4.jar:/home/ke/bin/tez/lib/commons-lang-2.6.jar:/home/ke/bin/tez/lib
/commons-lang3-3.7.jar:/home/ke/bin/tez/lib/commons-logging-1.1.3.jar:/home/ke/bin/tez/lib/commons-math3-3.1.1.jar:/home/ke/bin/tez/lib/commons-net-3.6.jar:/home/ke/bin/tez/lib/commons-text-1.4.jar:/home/ke/bin/tez/lib/curator-client-2.13.0.jar:/home/ke/bin/tez/lib/curator-framework-2.13.0.jar:/home/ke/bin/tez/lib/curator-recipes-2.13.0.jar:/home/ke/bin/tez/lib/dnsjava-2.1.7.jar:/home/ke/bin/tez/lib/ehcache-3.3.1.jar:/home/ke/bin/tez/lib/error_prone_annotations-2.2.0.jar:/home/ke/bin/tez/lib/failureaccess-1.0.jar:/home/ke/bin/tez/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/ke/bin/tez/lib/gson-2.2.4.jar:/home/ke/bin/tez/lib/guava-27.0-jre.jar:/home/ke/bin/tez/lib/guice-4.0.jar:/home/ke/bin/tez/lib/guice-servlet-4.0.jar:/home/ke/bin/tez/lib/hadoop-annotations-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-auth-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-hdfs-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-hdfs-client-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-map
reduce-client-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-mapreduce-client-core-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-api-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-client-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-registry-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-server-common-3.2.1.jar:/home/ke/bin/tez/lib/hadoop-yarn-server-web-proxy-3.2.1.jar:/home/ke/bin/tez/lib/HikariCP-java7-2.4.12.jar:/home/ke/bin/tez/lib/htrace-core4-4.1.0-incubating.jar:/home/ke/bin/tez/lib/httpclient-4.5.6.jar:/home/ke/bin/tez/lib/httpcore-4.4.10.jar:/home/ke/bin/tez/lib/j2objc-annotations-1.1.jar:/home/ke/bin/tez/lib/jackson-annotations-2.9.8.jar:/home/ke/bin/tez/lib/jackson-core-2.9.8.jar:/home/ke/bin/tez/lib/jackson-core-asl-1.9.2.jar:/home/ke/bin/tez/lib/jackson-databind-2.9.8.jar:/home/ke/bin/tez/lib/jackson-jaxrs-1.9.2.jar:/home/ke/bin/tez/lib/jackson-jaxrs-base-2.9.8.jar:/home/ke/bin/tez/lib/jackson-jaxrs-json-provider-2.9.8.jar:/home/ke/bin
/tez/lib/jackson-mapper-asl-1.9.2.jar:/home/ke/bin/tez/lib/jackson-module-jaxb-annotations-2.9.8.jar:/home/ke/bin/tez/lib/jackson-xc-1.9.2.jar:/home/ke/bin/tez/lib/javax.inject-1.jar:/home/ke/bin/tez/lib/javax.servlet-api-3.1.0.jar:/home/ke/bin/tez/lib/jaxb-api-2.2.11.jar:/home/ke/bin/tez/lib/jaxb-impl-2.2.3-1.jar:/home/ke/bin/tez/lib/jcip-annotations-1.0-1.jar:/home/ke/bin/tez/lib/jersey-client-1.19.jar:/home/ke/bin/tez/lib/jersey-core-1.19.jar:/home/ke/bin/tez/lib/jersey-guice-1.19.jar:/home/ke/bin/tez/lib/jersey-json-1.19.jar:/home/ke/bin/tez/lib/jersey-servlet-1.19.jar:/home/ke/bin/tez/lib/jettison-1.3.4.jar:/home/ke/bin/tez/lib/jetty-http-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-io-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-security-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-server-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-servlet-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-util-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-util-ajax-9.3.24.v2018060
5.jar:/home/ke/bin/tez/lib/jetty-webapp-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/jetty-xml-9.3.24.v20180605.jar:/home/ke/bin/tez/lib/json-smart-2.3.jar:/home/ke/bin/tez/lib/jsr305-3.0.0.jar:/home/ke/bin/tez/lib/jsr311-api-1.1.1.jar:/home/ke/bin/tez/lib/kerb-admin-1.0.1.jar:/home/ke/bin/tez/lib/kerb-client-1.0.1.jar:/home/ke/bin/tez/lib/kerb-common-1.0.1.jar:/home/ke/bin/tez/lib/kerb-core-1.0.1.jar:/home/ke/bin/tez/lib/kerb-crypto-1.0.1.jar:/home/ke/bin/tez/lib/kerb-identity-1.0.1.jar:/home/ke/bin/tez/lib/kerb-server-1.0.1.jar:/home/ke/bin/tez/lib/kerb-simplekdc-1.0.1.jar:/home/ke/bin/tez/lib/kerb-util-1.0.1.jar:/home/ke/bin/tez/lib/kerby-asn1-1.0.1.jar:/home/ke/bin/tez/lib/kerby-config-1.0.1.jar:/home/ke/bin/tez/lib/kerby-pkix-1.0.1.jar:/home/ke/bin/tez/lib/kerby-util-1.0.1.jar:/home/ke/bin/tez/lib/kerby-xdr-1.0.1.jar:/home/ke/bin/tez/lib/leveldbjni-all-1.8.jar:/home/ke/bin/tez/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/ke/bin/tez/lib/log4j-1.2.17.jar:
/home/ke/bin/tez/lib/metrics-core-3.1.0.jar:/home/ke/bin/tez/lib/mssql-jdbc-6.2.1.jre7.jar:/home/ke/bin/tez/lib/netty-3.6.2.Final.jar:/home/ke/bin/tez/lib/netty-all-4.0.52.Final.jar:/home/ke/bin/tez/lib/nimbus-jose-jwt-4.41.1.jar:/home/ke/bin/tez/lib/okhttp-2.7.5.jar:/home/ke/bin/tez/lib/okio-1.6.0.jar:/home/ke/bin/tez/lib/paranamer-2.3.jar:/home/ke/bin/tez/lib/protobuf-java-2.5.0.jar:/home/ke/bin/tez/lib/re2j-1.1.jar:/home/ke/bin/tez/lib/RoaringBitmap-0.5.21.jar:/home/ke/bin/tez/lib/servlet-api-2.5.jar:/home/ke/bin/tez/lib/slf4j-api-1.7.10.jar:/home/ke/bin/tez/lib/slf4j-log4j12-1.7.10.jar:/home/ke/bin/tez/lib/snappy-java-1.0.5.jar:/home/ke/bin/tez/lib/stax2-api-3.1.4.jar:/home/ke/bin/tez/lib/stax-api-1.0.1.jar:/home/ke/bin/tez/lib/token-provider-1.0.1.jar:/home/ke/bin/tez/lib/woodstox-core-5.0.3.jar:/home/ke/bin/tez/lib/zookeeper-3.4.13.jar:/home/ke/conf/devtest/client-devtest/bigdata/hive:$PWD/__spark_conf__/__hadoop_conf__"
# some key information in directory.info:
find -L . -maxdepth 5 -ls:
4294973945 0 drwx--s--- 3 bigdata hadoop 159 Nov 11 11:30 .
6442461027 0 drwx--s--- 2 bigdata hadoop 10 Nov 11 11:30 ./tmp
4294973950 24 -rwx------ 1 bigdata hadoop 22397 Nov 11 11:30 ./launch_container.sh
4294976100 4 -rw------- 1 bigdata hadoop 511 Nov 11 11:30 ./container_tokens
17179897731 0 drwx------ 3 bigdata bigdata 182 Nov 11 11:30 ./__spark_conf__
17179897732 4 -r-x------ 1 bigdata bigdata 2083 Nov 11 11:30 ./__spark_conf__/log4j.properties
19348229648 0 drwx------ 2 bigdata bigdata 223 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__
19348229379 4 -r-x------ 1 bigdata bigdata 3054 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
19348229412 4 -r-x------ 1 bigdata bigdata 702 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/log4j.properties
19348229422 16 -r-x------ 1 bigdata bigdata 14665 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
19348229430 4 -r-x------ 1 bigdata bigdata 3068 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/core-site.xml
19348228517 4 -r-x------ 1 bigdata bigdata 1382 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/switch_hadoop_env.sh
19348228540 16 -r-x------ 1 bigdata bigdata 12710 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
19348222919 184 -r-x------ 1 bigdata bigdata 184528 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/hive-site.xml
19348222920 8 -r-x------ 1 bigdata bigdata 8052 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/ozone-site.xml
17179897734 208 -r-x------ 1 bigdata bigdata 209545 Nov 11 11:30 ./__spark_conf__/__spark_hadoop_conf__.xml
17179897740 4 -r-x------ 1 bigdata bigdata 1138 Nov 11 11:30 ./__spark_conf__/__spark_conf__.properties
17179897744 4 -r-x------ 1 bigdata bigdata 617 Nov 11 11:30 ./__spark_conf__/__spark_dist_cache__.properties
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1308713042
I dont' know enough about the implications of this change to review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1311254076
@xkrogen Thanks for your review. In our cluster, YARN_CONF_DIR is same with HADOOP_CONF_DIR.
SparkHadoopUtil.newConfiguration is different from SparkHadoopUtil.get.newConfiguration
The steps in SparkHadoopUtil.newConfiguration are below:
*(step a) load configuration from config in classpath
*(step b) append s3 and spark.hadoop.* and spark.hive.* configuraiton
The steps in SparkHadoopUtil.get.newConfiguration are below:
*(step a) load configuration from config in classpath
*(step b) append s3 and spark.hadoop.* and spark.hive.* configuraiton
*(step c) append __spark_hadoop_conf__.xml configuration
I add some key information about the failed application.
```
# some key information in launch_container.sh:
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/XXX/conf/devtest/server/yarn/hadoop"}
export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/home/xxx/conf/devtest/client-devtest/bigdata/hadoop:/home/xxx/bin/hadoop/share/hadoop/common/lib/*:/home/xxx/bin/hadoop/share/hadoop/common/*:/home/xxx/bin/hadoop/share/hadoop/hdfs:/home/xxx/bin/hadoop/share/hadoop/hdfs/lib/*:/home/xxx/bin/hadoop/share/hadoop/hdfs/*:/home/xxx/bin/hadoop/share/hadoop/mapreduce/lib/*:/home/xxx/bin/hadoop/share/hadoop/mapreduce/*:/home/xxx/bin/hadoop/share/hadoop/yarn:/home/xxx/bin/hadoop/share/hadoop/yarn/lib/*:/home/xxx/bin/hadoop/share/hadoop/yarn/*:/home/xxx/conf/devtest/client-devtest/bigdata/tez:/home/xxx/bin/tez/hadoop-shim-0.
9.2.jar:/home/xxx/bin/tez/hadoop-shim-2.7-0.9.2.jar:/home/xxx/bin/tez/lib:/home/xxx/bin/tez/LICENSE:/home/xxx/bin/tez/LICENSE-BSD-3clause:/home/xxx/bin/tez/LICENSE-CDDLv1.0:/home/xxx/bin/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/home/xxx/bin/tez/LICENSE-MIT:/home/xxx/bin/tez/LICENSE-SIL_OpenFontLicense-v1.1:/home/xxx/bin/tez/NOTICE:/home/xxx/bin/tez/tez-api-0.9.2.jar:/home/xxx/bin/tez/tez-build-tools-0.9.2.jar:/home/xxx/bin/tez/tez-common-0.9.2.jar:/home/xxx/bin/tez/tez-dag-0.9.2.jar:/home/xxx/bin/tez/tez-examples-0.9.2.jar:/home/xxx/bin/tez/tez-ext-service-tests-0.9.2.jar:/home/xxx/bin/tez/tez-history-parser-0.9.2.jar:/home/xxx/bin/tez/tez-javadoc-tools-0.9.2.jar:/home/xxx/bin/tez/tez-job-analyzer-0.9.2.jar:/home/xxx/bin/tez/tez-mapreduce-0.9.2.jar:/home/xxx/bin/tez/tez-protobuf-history-plugin-0.9.2.jar:/home/xxx/bin/tez/tez-runtime-internals-0.9.2.jar:/home/xxx/bin/tez/tez-runtime-library-0.9.2.jar:/home/xxx/bin/tez/tez-tests-0.9.2.jar:/home/xxx/bin/tez/tez-ui-0.9.2.war:/home/xxx/bin/te
z/tez-yarn-timeline-history-0.9.2.jar:/home/xxx/bin/tez/tez-yarn-timeline-history-with-acls-0.9.2.jar:/home/xxx/bin/tez/lib/accessors-smart-1.2.jar:/home/xxx/bin/tez/lib/animal-sniffer-annotations-1.17.jar:/home/xxx/bin/tez/lib/aopalliance-1.0.jar:/home/xxx/bin/tez/lib/asm-5.0.4.jar:/home/xxx/bin/tez/lib/async-http-client-1.8.16.jar:/home/xxx/bin/tez/lib/audience-annotations-0.5.0.jar:/home/xxx/bin/tez/lib/avro-1.7.7.jar:/home/xxx/bin/tez/lib/bcpkix-jdk15on-1.60.jar:/home/xxx/bin/tez/lib/bcprov-jdk15on-1.60.jar:/home/xxx/bin/tez/lib/checker-qual-2.5.2.jar:/home/xxx/bin/tez/lib/commons-beanutils-1.9.3.jar:/home/xxx/bin/tez/lib/commons-cli-1.2.jar:/home/xxx/bin/tez/lib/commons-codec-1.4.jar:/home/xxx/bin/tez/lib/commons-collections-3.2.2.jar:/home/xxx/bin/tez/lib/commons-collections4-4.1.jar:/home/xxx/bin/tez/lib/commons-compress-1.18.jar:/home/xxx/bin/tez/lib/commons-configuration2-2.1.1.jar:/home/xxx/bin/tez/lib/commons-daemon-1.0.13.jar:/home/xxx/bin/tez/lib/commons-io-2.4.jar:/hom
e/xxx/bin/tez/lib/commons-lang-2.6.jar:/home/xxx/bin/tez/lib/commons-lang3-3.7.jar:/home/xxx/bin/tez/lib/commons-logging-1.1.3.jar:/home/xxx/bin/tez/lib/commons-math3-3.1.1.jar:/home/xxx/bin/tez/lib/commons-net-3.6.jar:/home/xxx/bin/tez/lib/commons-text-1.4.jar:/home/xxx/bin/tez/lib/curator-client-2.13.0.jar:/home/xxx/bin/tez/lib/curator-framework-2.13.0.jar:/home/xxx/bin/tez/lib/curator-recipes-2.13.0.jar:/home/xxx/bin/tez/lib/dnsjava-2.1.7.jar:/home/xxx/bin/tez/lib/ehcache-3.3.1.jar:/home/xxx/bin/tez/lib/error_prone_annotations-2.2.0.jar:/home/xxx/bin/tez/lib/failureaccess-1.0.jar:/home/xxx/bin/tez/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/xxx/bin/tez/lib/gson-2.2.4.jar:/home/xxx/bin/tez/lib/guava-27.0-jre.jar:/home/xxx/bin/tez/lib/guice-4.0.jar:/home/xxx/bin/tez/lib/guice-servlet-4.0.jar:/home/xxx/bin/tez/lib/hadoop-annotations-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-auth-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-hdfs-3.2.1.jar:/
home/xxx/bin/tez/lib/hadoop-hdfs-client-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-mapreduce-client-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-mapreduce-client-core-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-api-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-client-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-registry-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-server-common-3.2.1.jar:/home/xxx/bin/tez/lib/hadoop-yarn-server-web-proxy-3.2.1.jar:/home/xxx/bin/tez/lib/HikariCP-java7-2.4.12.jar:/home/xxx/bin/tez/lib/htrace-core4-4.1.0-incubating.jar:/home/xxx/bin/tez/lib/httpclient-4.5.6.jar:/home/xxx/bin/tez/lib/httpcore-4.4.10.jar:/home/xxx/bin/tez/lib/j2objc-annotations-1.1.jar:/home/xxx/bin/tez/lib/jackson-annotations-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-core-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-core-asl-1.9.2.jar:/home/xxx/bin/tez/lib/jackson-databind-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-jaxrs-1.9.2.jar:/home/xxx/bin/tez/lib/
jackson-jaxrs-base-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-jaxrs-json-provider-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-mapper-asl-1.9.2.jar:/home/xxx/bin/tez/lib/jackson-module-jaxb-annotations-2.9.8.jar:/home/xxx/bin/tez/lib/jackson-xc-1.9.2.jar:/home/xxx/bin/tez/lib/javax.inject-1.jar:/home/xxx/bin/tez/lib/javax.servlet-api-3.1.0.jar:/home/xxx/bin/tez/lib/jaxb-api-2.2.11.jar:/home/xxx/bin/tez/lib/jaxb-impl-2.2.3-1.jar:/home/xxx/bin/tez/lib/jcip-annotations-1.0-1.jar:/home/xxx/bin/tez/lib/jersey-client-1.19.jar:/home/xxx/bin/tez/lib/jersey-core-1.19.jar:/home/xxx/bin/tez/lib/jersey-guice-1.19.jar:/home/xxx/bin/tez/lib/jersey-json-1.19.jar:/home/xxx/bin/tez/lib/jersey-servlet-1.19.jar:/home/xxx/bin/tez/lib/jettison-1.3.4.jar:/home/xxx/bin/tez/lib/jetty-http-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-io-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-security-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-server-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-servlet-9.3.24
.v20180605.jar:/home/xxx/bin/tez/lib/jetty-util-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-util-ajax-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-webapp-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/jetty-xml-9.3.24.v20180605.jar:/home/xxx/bin/tez/lib/json-smart-2.3.jar:/home/xxx/bin/tez/lib/jsr305-3.0.0.jar:/home/xxx/bin/tez/lib/jsr311-api-1.1.1.jar:/home/xxx/bin/tez/lib/kerb-admin-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-client-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-common-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-core-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-crypto-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-identity-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-server-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-simplekdc-1.0.1.jar:/home/xxx/bin/tez/lib/kerb-util-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-asn1-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-config-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-pkix-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-util-1.0.1.jar:/home/xxx/bin/tez/lib/kerby-xdr-1.0.1.jar:/home/xxx/bin/tez/lib/lev
eldbjni-all-1.8.jar:/home/xxx/bin/tez/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/xxx/bin/tez/lib/log4j-1.2.17.jar:/home/xxx/bin/tez/lib/metrics-core-3.1.0.jar:/home/xxx/bin/tez/lib/mssql-jdbc-6.2.1.jre7.jar:/home/xxx/bin/tez/lib/netty-3.6.2.Final.jar:/home/xxx/bin/tez/lib/netty-all-4.0.52.Final.jar:/home/xxx/bin/tez/lib/nimbus-jose-jwt-4.41.1.jar:/home/xxx/bin/tez/lib/okhttp-2.7.5.jar:/home/xxx/bin/tez/lib/okio-1.6.0.jar:/home/xxx/bin/tez/lib/paranamer-2.3.jar:/home/xxx/bin/tez/lib/protobuf-java-2.5.0.jar:/home/xxx/bin/tez/lib/re2j-1.1.jar:/home/xxx/bin/tez/lib/RoaringBitmap-0.5.21.jar:/home/xxx/bin/tez/lib/servlet-api-2.5.jar:/home/xxx/bin/tez/lib/slf4j-api-1.7.10.jar:/home/xxx/bin/tez/lib/slf4j-log4j12-1.7.10.jar:/home/xxx/bin/tez/lib/snappy-java-1.0.5.jar:/home/xxx/bin/tez/lib/stax2-api-3.1.4.jar:/home/xxx/bin/tez/lib/stax-api-1.0.1.jar:/home/xxx/bin/tez/lib/token-provider-1.0.1.jar:/home/xxx/bin/tez/lib/woodstox-core-5.0.3.jar:/home/xxx/bin/tez/lib/
zookeeper-3.4.13.jar:/home/xxx/conf/devtest/client-devtest/bigdata/hive:$PWD/__spark_conf__/__hadoop_conf__"
# some key information in directory.info:
find -L . -maxdepth 5 -ls:
4294973945 0 drwx--s--- 3 bigdata hadoop 159 Nov 11 11:30 .
6442461027 0 drwx--s--- 2 bigdata hadoop 10 Nov 11 11:30 ./tmp
4294973950 24 -rwx------ 1 bigdata hadoop 22397 Nov 11 11:30 ./launch_container.sh
4294976100 4 -rw------- 1 bigdata hadoop 511 Nov 11 11:30 ./container_tokens
17179897731 0 drwx------ 3 bigdata bigdata 182 Nov 11 11:30 ./__spark_conf__
17179897732 4 -r-x------ 1 bigdata bigdata 2083 Nov 11 11:30 ./__spark_conf__/log4j.properties
19348229648 0 drwx------ 2 bigdata bigdata 223 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__
19348229379 4 -r-x------ 1 bigdata bigdata 3054 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/mapred-site.xml
19348229412 4 -r-x------ 1 bigdata bigdata 702 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/log4j.properties
19348229422 16 -r-x------ 1 bigdata bigdata 14665 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/yarn-site.xml
19348229430 4 -r-x------ 1 bigdata bigdata 3068 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/core-site.xml
19348228517 4 -r-x------ 1 bigdata bigdata 1382 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/switch_hadoop_env.sh
19348228540 16 -r-x------ 1 bigdata bigdata 12710 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml
19348222919 184 -r-x------ 1 bigdata bigdata 184528 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/hive-site.xml
19348222920 8 -r-x------ 1 bigdata bigdata 8052 Nov 11 11:30 ./__spark_conf__/__hadoop_conf__/ozone-site.xml
17179897734 208 -r-x------ 1 bigdata bigdata 209545 Nov 11 11:30 ./__spark_conf__/__spark_hadoop_conf__.xml
17179897740 4 -r-x------ 1 bigdata bigdata 1138 Nov 11 11:30 ./__spark_conf__/__spark_conf__.properties
17179897744 4 -r-x------ 1 bigdata bigdata 617 Nov 11 11:30 ./__spark_conf__/__spark_dist_cache__.properties
```
In step a, load config from classpath. Here classpath contains HADOOP_CONF_DIR and \_\_spark_conf\_\_/\_\_hadoop_conf\_\_. HADOOP_CONF_DIR happend before \_\_spark_conf\_\_/\_\_hadoop_conf\_\_ , and HADOOP_CONF_DIR is the conf dir of nodemanager. so \_\_spark_conf\_\_/\_\_hadoop_conf\_\_/yarn-site.xml will be ignored.
So In this PR, I load \_\_spark_conf\_\_/__spark_hadoop_conf__.xml so that I can reload config from client.
> Note: It happen in our yarn federation cluster, because the client config is different from nodemanager client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1308320736
@dongjoon-hyun @srowen Can you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu closed pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
URL: https://github.com/apache/spark/pull/37949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1252898141
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
xkrogen commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1314492128
Ah, I see. It seems you're using `spark.yarn.populateHadoopClasspath = true`. It looks like it's expected that the Hadoop conf from the node overrides the one from `__hadoop_conf__` in this case:
https://github.com/apache/spark/blob/5600bef0ee6149ebc1abcf4c9c9b2991553ca3de/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1530-L1534
Seems this was done as part of #18370, maybe you can check there for context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu commented on pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on PR #37949:
URL: https://github.com/apache/spark/pull/37949#issuecomment-1314728385
@xkrogen Sorry for miss configuration 'spark.yarn.populateHadoopClasspath'. Thank you very much! When spark.yarn.populateHadoopClasspath is false, HADOOP_CONF_DIR will be removed in CLASSPATH. This config solve my problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengchenyu closed pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
Posted by GitBox <gi...@apache.org>.
zhengchenyu closed pull request #37949: [SPARK-40504][YARN] Make yarn appmaster load config from client
URL: https://github.com/apache/spark/pull/37949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org