You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Chaos-Ju <gi...@git.apache.org> on 2017/09/12 13:44:58 UTC
[GitHub] spark pull request #19206: Client and ApplicationMaster resolvePath is inapp...
GitHub user Chaos-Ju opened a pull request:
https://github.com/apache/spark/pull/19206
Client and ApplicationMaster resolvePath is inappropriate when use viewfs
## What changes were proposed in this pull request?
When HDFS use viewfs and spark construct Executor's and ApplicationMaster's localResource Map ( the list of localized files ) ,can't covert viewfs:// path to the real hdfs:// path . Therefore , when NodeManager download the local Resource, will throw java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://clusterName/
Exception stack：
java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://ns-view/
at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337)
at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:167)
at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:167)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Failing this attempt. Failing the application
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Chaos-Ju/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19206
----
commit f1fff009d32b8f7d1d2b24734e4d677c6264ec90
Author: Chaos-Ju <ch...@163.com>
Date: 2017-09-12T12:45:36Z
fix spark support viewfs
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19206
I also meant that I don't think this is a problem and the "fix" is actually incorrect. I disagree with "this is a feature", because if the path resolves to an HA namespace, and YARN doesn't know about the namespace, things will still not work. If the path resolves to a specific namenode in an HA setup, and that namenode is the standby one, it will still not work. So YARN needs to know about the HA/federation configuration for this feature to work.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by lw-lin <gi...@git.apache.org>.
Github user lw-lin commented on the issue:
https://github.com/apache/spark/pull/19206
It says `Spark-19206` in your PR title but Spark-19206 is actually about `Update outdated parameter descriptions in external-kafka module`; so maybe you should reference a different JIRA. That's all what Sean and Marcelo meant.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by Chaos-Ju <gi...@git.apache.org>.
Github user Chaos-Ju commented on the issue:
https://github.com/apache/spark/pull/19206
@srowen you think should close the jira and this is pointless？
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19206
Also, doesn't this mean that your YARN service does not know about the federated HDFS service?
Same thing can happen if you instead have an HA setup but YARN doesn't know about it. It will fail to resolve the namespace and YARN won't be able to download resources.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19206: [SPARK-19206][yarn]Client and ApplicationMaster r...
Posted by Chaos-Ju <gi...@git.apache.org>.
Github user Chaos-Ju closed the pull request at:
https://github.com/apache/spark/pull/19206
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: Client and ApplicationMaster resolvePath is inappropriat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19206
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19206
@Chaos-Ju this is connected to the wrong JIRA
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by Chaos-Ju <gi...@git.apache.org>.
Github user Chaos-Ju commented on the issue:
https://github.com/apache/spark/pull/19206
@vanzin @srowen thanks your notice and comment， I means that if make Yarn Cluster know about the federated HDFS service, We should distribute the mount table XML in the cluster and by restart Yarn Cluster bring into effect. under this circumstances，if We change the mount table or HA settings，we should restart Yarn Cluster again. In addition，I dont think this is a bug，but its a feature . if it is possible，We can make Yarn no need to know about the federated HDFS service like MR & Hive
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19206: [SPARK-19206][yarn]Client and ApplicationMaster resolveP...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19206
@Chaos-Ju Sean means the bug you reference in the title has nothing to do with the code.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org