You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2013/11/09 22:48:07 UTC
[4/4] git commit: Merge pull request #149 from
tgravescs/fixSecureHdfsAccess
Merge pull request #149 from tgravescs/fixSecureHdfsAccess
Fix secure hdfs access for spark on yarn
https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.
The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/8af99f23
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/8af99f23
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/8af99f23
Branch: refs/heads/master
Commit: 8af99f2356ed19fe43d722ada02f8802cbd46d40
Parents: 72a601e 13a1950
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Authored: Sat Nov 9 13:48:00 2013 -0800
Committer: Matei Zaharia <ma...@eecs.berkeley.edu>
Committed: Sat Nov 9 13:48:00 2013 -0800
----------------------------------------------------------------------
.../apache/spark/deploy/SparkHadoopUtil.scala | 21 +++++++++++++++-----
.../scala/org/apache/spark/rdd/HadoopRDD.scala | 2 ++
2 files changed, 18 insertions(+), 5 deletions(-)
----------------------------------------------------------------------