You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Suhas Satish <su...@gmail.com> on 2014/02/07 03:20:04 UTC

spark 0.9.0 compatible with hadoop 1.0.4 ?

Is spark compatible with Hadoop 1.0.4 ?

I built spark with the variable -
SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly

But I got the following jar -
spark-assembly-0.9.0-incubating-hadoop1.0.4.jar

When I ran a spark job, I'm  running  into this issue-

 ./pyspark ~/test/test_spark/spark.script
Found multiple Spark assembly jars in
/home/suhash/git_repos/spark-apache-v0.9.0-incubating/assembly/target/scala-2.10:
Traceback (most recent call last):
  File "/home/ssatish/test/test_spark/spark.script", line 2, in <module>
    sc=SparkContext('local', 'App Name')
  File
"/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/context.py",
line 83, in __init__
    SparkContext._ensure_initialized(self)
  File
"/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/context.py",
line 159, in _ensure_initialized
    SparkContext._gateway = launch_gateway()
  File
"/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/java_gateway.py",
line 46, in launch_gateway
    port = int(proc.stdout.readline())
ValueError: invalid literal for int() with base 10:
'spark-assembly-0.9.0-incubating-hadoop1.0.4.jar\n'
Exception AttributeError: "'SparkContext' object has no attribute '_jsc'"
in <bound method SparkContext.__del__ of <pyspark.context.SparkContext
object at 0x15eaa90>> ignored




Thanks,
Suhas.

Re: spark 0.9.0 compatible with hadoop 1.0.4 ?

Posted by Patrick Wendell <pw...@gmail.com>.

It looks like you have multiple copies of the assembly jar (see the
error at the top). Try doing a clean before you build

sbt/sbt clean assembly

On Thu, Feb 6, 2014 at 6:20 PM, Suhas Satish <su...@gmail.com> wrote:
> Is spark compatible with Hadoop 1.0.4 ?
>
> I built spark with the variable -
> SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly
>
> But I got the following jar -
> spark-assembly-0.9.0-incubating-hadoop1.0.4.jar
>
> When I ran a spark job, I'm  running  into this issue-
>
>  ./pyspark ~/test/test_spark/spark.script
> Found multiple Spark assembly jars in
> /home/suhash/git_repos/spark-apache-v0.9.0-incubating/assembly/target/scala-2.10:
> Traceback (most recent call last):
>   File "/home/ssatish/test/test_spark/spark.script", line 2, in <module>
>     sc=SparkContext('local', 'App Name')
>   File
> "/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/context.py",
> line 83, in __init__
>     SparkContext._ensure_initialized(self)
>   File
> "/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/context.py",
> line 159, in _ensure_initialized
>     SparkContext._gateway = launch_gateway()
>   File
> "/home/suhash/git_repos/spark-apache-v0.9.0-incubating/python/pyspark/java_gateway.py",
> line 46, in launch_gateway
>     port = int(proc.stdout.readline())
> ValueError: invalid literal for int() with base 10:
> 'spark-assembly-0.9.0-incubating-hadoop1.0.4.jar\n'
> Exception AttributeError: "'SparkContext' object has no attribute '_jsc'" in
> <bound method SparkContext.__del__ of <pyspark.context.SparkContext object
> at 0x15eaa90>> ignored
>
>
>
>
> Thanks,
> Suhas.
>