You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sunita Arvind <su...@gmail.com> on 2014/11/26 07:54:24 UTC

Spark setup on local windows machine

Hi All,

I just installed a spark on my laptop and trying to get spark-shell to
work. Here is the error I see:

C:\spark\bin>spark-shell
Exception in thread "main" java.util.NoSuchElementException: key not found:
CLAS
SPATH
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:58)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:58)
        at
org.apache.spark.deploy.SparkSubmitDriverBootstrapper$.main(SparkSubm
itDriverBootstrapper.scala:49)
        at
org.apache.spark.deploy.SparkSubmitDriverBootstrapper.main(SparkSubmi
tDriverBootstrapper.scala)


The classpath seems to be right:

C:\spark\bin>compute-classpath.cmd
;;C:\spark\bin\..\conf;C:\spark\bin\..\lib\spark-assembly-1.1.0-hadoop2.3.0.jar;
;C:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;C:\spark\bin\..\lib\datanucle
us-core-3.2.2.jar;C:\spark\bin\..\lib\datanucleus-rdbms-3.2.1.jar

Manually exporting the classpath to include the assembly jar doesnt help
either.

What could be wrong with this installation? Scala and SBT are installed, in
path and are working fine.

Appreciate your help.
regards
Sunita

Re: Spark setup on local windows machine

Posted by Sunita Arvind <su...@gmail.com>.

Thanks Sameer and Akhil for your help.

I tried both your suggestions however, I still face the same issue. There
was indeed space in the installation path for Scala and Sbt since I had let
the defaults stay and hence the path was C:\Program Files.... . I
reinstalled scala and sbt in c:\ as well as spark.
The spark binaries I am using are built internally from source by our team
and the same binary works for rest of the team.

Here is what the spark-env.cmd looks like for me:
set SCALA_HOME=C:\scala
set
SPARK_CLASSPATH=C:\spark\bin\..\conf;C:\spark\bin\..\lib\spark-assembly-1.1.0-hadoop2.3.0.jar;C:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;C:\spark\bin\..\lib\datanucleus-core-3.2.2.jar;C:\spark\bin\..\lib\datanucleus-rdbms-3.2.1.jar;C:\scala\lib\scala-library.jar;C:\scala\lib\scala-compiler.jar;



I get the same error inspite of this. The compute-classpath.cmd yields
correct results:

C:\spark>bin\spark-shell
Exception in thread "main" java.util.NoSuchElementException: key not found:
CLAS
SPATH
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:58)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:58)
        at
org.apache.spark.deploy.SparkSubmitDriverBootstrapper$.main(SparkSubm
itDriverBootstrapper.scala:49)
        at
org.apache.spark.deploy.SparkSubmitDriverBootstrapper.main(SparkSubmi
tDriverBootstrapper.scala)

regards
Sunita

On Tue, Nov 25, 2014 at 11:47 PM, Sameer Farooqui <sa...@databricks.com>
wrote:

> Hi Sunita,
>
> This gitbook may also be useful for you to get Spark running in local mode
> on your Windows machine:
> http://blueplastic.gitbooks.io/how-to-light-your-spark-on-a-stick/content/
>
> On Tue, Nov 25, 2014 at 11:09 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> You could try following this guidelines
>> http://docs.sigmoidanalytics.com/index.php/How_to_build_SPARK_on_Windows
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Nov 26, 2014 at 12:24 PM, Sunita Arvind <su...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I just installed a spark on my laptop and trying to get spark-shell to
>>> work. Here is the error I see:
>>>
>>> C:\spark\bin>spark-shell
>>> Exception in thread "main" java.util.NoSuchElementException: key not
>>> found: CLAS
>>> SPATH
>>>         at scala.collection.MapLike$class.default(MapLike.scala:228)
>>>         at scala.collection.AbstractMap.default(Map.scala:58)
>>>         at scala.collection.MapLike$class.apply(MapLike.scala:141)
>>>         at scala.collection.AbstractMap.apply(Map.scala:58)
>>>         at
>>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper$.main(SparkSubm
>>> itDriverBootstrapper.scala:49)
>>>         at
>>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper.main(SparkSubmi
>>> tDriverBootstrapper.scala)
>>>
>>>
>>> The classpath seems to be right:
>>>
>>> C:\spark\bin>compute-classpath.cmd
>>>
>>> ;;C:\spark\bin\..\conf;C:\spark\bin\..\lib\spark-assembly-1.1.0-hadoop2.3.0.jar;
>>>
>>> ;C:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;C:\spark\bin\..\lib\datanucle
>>> us-core-3.2.2.jar;C:\spark\bin\..\lib\datanucleus-rdbms-3.2.1.jar
>>>
>>> Manually exporting the classpath to include the assembly jar doesnt help
>>> either.
>>>
>>> What could be wrong with this installation? Scala and SBT are installed,
>>> in path and are working fine.
>>>
>>> Appreciate your help.
>>> regards
>>> Sunita
>>>
>>>
>>>
>>
>

Re: Spark setup on local windows machine

Posted by Sameer Farooqui <sa...@databricks.com>.

Hi Sunita,

This gitbook may also be useful for you to get Spark running in local mode
on your Windows machine:
http://blueplastic.gitbooks.io/how-to-light-your-spark-on-a-stick/content/

On Tue, Nov 25, 2014 at 11:09 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> You could try following this guidelines
> http://docs.sigmoidanalytics.com/index.php/How_to_build_SPARK_on_Windows
>
> Thanks
> Best Regards
>
> On Wed, Nov 26, 2014 at 12:24 PM, Sunita Arvind <su...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I just installed a spark on my laptop and trying to get spark-shell to
>> work. Here is the error I see:
>>
>> C:\spark\bin>spark-shell
>> Exception in thread "main" java.util.NoSuchElementException: key not
>> found: CLAS
>> SPATH
>>         at scala.collection.MapLike$class.default(MapLike.scala:228)
>>         at scala.collection.AbstractMap.default(Map.scala:58)
>>         at scala.collection.MapLike$class.apply(MapLike.scala:141)
>>         at scala.collection.AbstractMap.apply(Map.scala:58)
>>         at
>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper$.main(SparkSubm
>> itDriverBootstrapper.scala:49)
>>         at
>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper.main(SparkSubmi
>> tDriverBootstrapper.scala)
>>
>>
>> The classpath seems to be right:
>>
>> C:\spark\bin>compute-classpath.cmd
>>
>> ;;C:\spark\bin\..\conf;C:\spark\bin\..\lib\spark-assembly-1.1.0-hadoop2.3.0.jar;
>>
>> ;C:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;C:\spark\bin\..\lib\datanucle
>> us-core-3.2.2.jar;C:\spark\bin\..\lib\datanucleus-rdbms-3.2.1.jar
>>
>> Manually exporting the classpath to include the assembly jar doesnt help
>> either.
>>
>> What could be wrong with this installation? Scala and SBT are installed,
>> in path and are working fine.
>>
>> Appreciate your help.
>> regards
>> Sunita
>>
>>
>>
>

Re: Spark setup on local windows machine

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

You could try following this guidelines
http://docs.sigmoidanalytics.com/index.php/How_to_build_SPARK_on_Windows

Thanks
Best Regards

On Wed, Nov 26, 2014 at 12:24 PM, Sunita Arvind <su...@gmail.com>
wrote:

> Hi All,
>
> I just installed a spark on my laptop and trying to get spark-shell to
> work. Here is the error I see:
>
> C:\spark\bin>spark-shell
> Exception in thread "main" java.util.NoSuchElementException: key not
> found: CLAS
> SPATH
>         at scala.collection.MapLike$class.default(MapLike.scala:228)
>         at scala.collection.AbstractMap.default(Map.scala:58)
>         at scala.collection.MapLike$class.apply(MapLike.scala:141)
>         at scala.collection.AbstractMap.apply(Map.scala:58)
>         at
> org.apache.spark.deploy.SparkSubmitDriverBootstrapper$.main(SparkSubm
> itDriverBootstrapper.scala:49)
>         at
> org.apache.spark.deploy.SparkSubmitDriverBootstrapper.main(SparkSubmi
> tDriverBootstrapper.scala)
>
>
> The classpath seems to be right:
>
> C:\spark\bin>compute-classpath.cmd
>
> ;;C:\spark\bin\..\conf;C:\spark\bin\..\lib\spark-assembly-1.1.0-hadoop2.3.0.jar;
>
> ;C:\spark\bin\..\lib\datanucleus-api-jdo-3.2.1.jar;C:\spark\bin\..\lib\datanucle
> us-core-3.2.2.jar;C:\spark\bin\..\lib\datanucleus-rdbms-3.2.1.jar
>
> Manually exporting the classpath to include the assembly jar doesnt help
> either.
>
> What could be wrong with this installation? Scala and SBT are installed,
> in path and are working fine.
>
> Appreciate your help.
> regards
> Sunita
>
>
>