You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/06/22 16:40:57 UTC
[jira] [Updated] (SPARK-16145) spark-ec2 script on 1.6.1 does allow
instances to use sqlContext
[ https://issues.apache.org/jira/browse/SPARK-16145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-16145:
------------------------------
Priority: Major (was: Blocker)
I don't think the EC2 script is supported anymore, certainly not for 2.x.
> spark-ec2 script on 1.6.1 does allow instances to use sqlContext
> ----------------------------------------------------------------
>
> Key: SPARK-16145
> URL: https://issues.apache.org/jira/browse/SPARK-16145
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Affects Versions: 1.6.1
> Environment: AWS EC2
> Reporter: Rick Bross
>
> Downloaded 1.6.1 for Hadoop 2.4.
> I used the spark-ec2 script to create a cluster and I'm running into an issue which prevents importing sqlContext. Reading prior reports I looked at the output to find the first error:
> java.lang.RuntimeException: java.io.IOException: Filesystem closed
> Not sure how to diagnose this. Exiting the Spark REPL and reentering, every subsequent time I get this error:
> java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx-x-x
> I assume that some env script is specifying this, since /tmp/hive doesn't exist. I thought that this would be taken care of by the spark-ec2 script so you could just go to town.
> I have no experience with HDFS. I have used Spark on Cassandra and on of S3, but I've never deployed it myself. I tried this:
> root@ip-172-31-57-109 ephemeral-hdfs]$ bin/hadoop fs -ls
> Warning: $HADOOP_HOME is deprecated.
> ls: Cannot access .: No such file or directory.
> I did see that under /mnt there is the ephemeral-hdfs folder which is in core-site.xml, but there is no tmp folder.
> I tried again with the download for Hadoop 1.x.
> Same behavior.
> It's curious to me that spark-ec2 has an argument for specifying the Hadoop version; is this required? It would seem that you've already specified it when downloading.
> I tried to create the path "tmp/hive" under /mnt/ephemeral-hdfs and chmod to 777. No joy.
> sqlContext is obviously a critical part of the Spark platform. The interesting thing is that I don't need HDFS at all - I'm going to be reading from S3 and writing to MySQL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org