You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Luca Venturini (JIRA)" <ji...@apache.org> on 2015/01/21 13:29:34 UTC
[jira] [Commented] (SPARK-2356) Exception: Could not locate
executable null\bin\winutils.exe in the Hadoop
[ https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285563#comment-14285563 ]
Luca Venturini commented on SPARK-2356:
---------------------------------------
This error occurs also within the spark.shell and mllib examples, where the execution simply stops.
The abovementioned workaround works by setting a windows environment variable called HADOOP_HOME .
> Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
> ---------------------------------------------------------------------------
>
> Key: SPARK-2356
> URL: https://issues.apache.org/jira/browse/SPARK-2356
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Kostiantyn Kudriavtsev
> Priority: Critical
>
> I'm trying to run some transformation on Spark, it works fine on cluster (YARN, linux machines). However, when I'm trying to run it on local machine (Windows 7) under unit test, I got errors (I don't use Hadoop, I'm read file from local filesystem):
> {code}
> 14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
> at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
> at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
> at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
> at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:36)
> at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:109)
> at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:228)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:97)
> {code}
> It's happened because Hadoop config is initialized each time when spark context is created regardless is hadoop required or not.
> I propose to add some special flag to indicate if hadoop config is required (or start this configuration manually)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org