[jira] [Created] (SPARK-16145) spark-ec2 script on 1.6.1 does allow instances to use sqlContext

["Rick Bross (JIRA)" <ji...@apache.org>]

Rick Bross created SPARK-16145:
----------------------------------

             Summary: spark-ec2 script on 1.6.1 does allow instances to use sqlContext
                 Key: SPARK-16145
                 URL: https://issues.apache.org/jira/browse/SPARK-16145
             Project: Spark
          Issue Type: Bug
          Components: EC2
    Affects Versions: 1.6.1
         Environment: AWS EC2
            Reporter: Rick Bross
            Priority: Blocker


Downloaded 1.6.1 for Hadoop 2.4.

I used the spark-ec2 script to create a cluster and I'm running into an issue which prevents importing sqlContext.  Reading prior reports I looked at the output to find the first error:

java.lang.RuntimeException: java.io.IOException: Filesystem closed

Not sure how to diagnose this.  Exiting the Spark REPL and reentering, every subsequent time I get this error:

java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx-x-x

I assume that some env script is specifying this, since /tmp/hive doesn't exist. I thought that this would be taken care of by the spark-ec2 script so you could just go to town.

I have no experience with HDFS. I have used  Spark on Cassandra and on of S3, but I've never deployed it myself. I tried this:

root@ip-172-31-57-109 ephemeral-hdfs]$ bin/hadoop fs -ls

Warning: $HADOOP_HOME is deprecated.

ls: Cannot access .: No such file or directory.

I did see that under /mnt there is the ephemeral-hdfs folder which is in core-site.xml, but there is no tmp folder. 

I tried again with the download for Hadoop 1.x.

Same behavior.

It's curious to me that spark-ec2 has an argument for specifying the Hadoop version; is this required? It would seem that you've already specified it when downloading.

I tried to create the path "tmp/hive" under /mnt/ephemeral-hdfs and chmod to 777. No joy.

sqlContext is obviously a critical part of the Spark platform.  The interesting thing is that I don't need HDFS at all - I'm going to be reading from S3 and writing to MySQL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org