You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/02/13 19:45:54 UTC
[GitHub] skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support
client dependencies with a Hadoop Compatible File System
skonto edited a comment on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System
URL: https://github.com/apache/spark/pull/23546#issuecomment-463340476
@vanzin I am calling you here because I saw you did the work with adding net auth secret support to the k8s side.
The integration test `Run SparkRemoteFileTest using a remote data file` was failing above due to:
```
2019-02-13 18:21:18 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.$anonfun$getSecretKey$6(SecurityManager.scala:328)
at scala.Option.getOrElse(Option.scala:138)
at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:327)
at org.apache.spark.util.Utils$.constructURIForAuthentication(Utils.scala:378)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:658)
at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:138)
```
That is caused because early in spark-submit I use `downloadFileList` that calls doFetchFile at some point
and that point the driver has not generated the key as secMgr is not initialized.
I noticed that Security manager's `initializeAuth()` is called from SparkEnv set in SparkContext.
The problem is that a few lines before my call to `downloadFileList` there are [several similar calls](https://github.com/apache/spark/blob/a829234df35c87c169425f2c79fd1963b5420888/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L367-L376) for client mode.
I managed to work-around this by filtering out http(s) urls as I dont need to touch them but the other calls will probably fail... thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org