You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by gaurav pathak <ga...@gmail.com> on 2016/02/29 14:27:45 UTC

Spark on Windows platform

Can someone guide me the steps and information regarding, installation of
SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be
great to read your experiences in using SPARK on Windows platform.


Thanks & Regards,
Gaurav Pathak

Re: Spark on Windows platform

Posted by Steve Loughran <st...@hortonworks.com>.

On 29 Feb 2016, at 13:40, gaurav pathak <ga...@gmail.com>> wrote:


Thanks Jorn.

Any guidance on how to get started with getting SPARK on Windows, is highly appreciated.

Thanks & Regards

Gaurav Pathak


you are at risk of seeing stack traces when you try to talk to the local filesystem, on account of (a) hadoop being part of the process and (b) it needing some native windows binaries

details: https://wiki.apache.org/hadoop/WindowsProblems

those binaries: https://github.com/steveloughran/winutils

(I need to add some 2.7.2 binaries in there, by the look of things)

Re: Spark on Windows platform

Posted by Gaurav Agarwal <ga...@gmail.com>.

> Hi
> I am running spark on windows but a standalone one.
>
> Use this code
>
> SparkConf conf = new
SparkConf().setMaster("local[1]").seatAppName("spark").setSparkHome("c:/spark/bin/spark-submit.cmd");
>
> Where sparkhome is the path where u extracted ur spark binaries till
bin/*.cmd
>
> You will get spark context or streaming context
>
> Thanks
>
> On Feb 29, 2016 7:10 PM, "gaurav pathak" <ga...@gmail.com>
wrote:
>>
>> Thanks Jorn.
>>
>> Any guidance on how to get started with getting SPARK on Windows, is
highly appreciated.
>>
>> Thanks & Regards
>>
>> Gaurav Pathak
>>
>> ~ sent from handheld device
>>
>> On Feb 29, 2016 5:34 AM, "Jörn Franke" <jo...@gmail.com> wrote:
>>>
>>> I think Hortonworks has a Windows Spark distribution. Maybe Bigtop as
well?
>>>
>>> > On 29 Feb 2016, at 14:27, gaurav pathak <ga...@gmail.com>
wrote:
>>> >
>>> > Can someone guide me the steps and information regarding,
installation of SPARK on Windows 7/8.1/10 , as well as on Windows Server.
Also, it will be great to read your experiences in using SPARK on Windows
platform.
>>> >
>>> >
>>> > Thanks & Regards,
>>> > Gaurav Pathak

Re: Spark on Windows platform

Posted by gaurav pathak <ga...@gmail.com>.

Thanks Jorn.

Any guidance on how to get started with getting SPARK on Windows, is highly
appreciated.

Thanks & Regards

Gaurav Pathak

~ sent from handheld device
On Feb 29, 2016 5:34 AM, "Jörn Franke" <jo...@gmail.com> wrote:

> I think Hortonworks has a Windows Spark distribution. Maybe Bigtop as well?
>
> > On 29 Feb 2016, at 14:27, gaurav pathak <ga...@gmail.com>
> wrote:
> >
> > Can someone guide me the steps and information regarding, installation
> of SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will
> be great to read your experiences in using SPARK on Windows platform.
> >
> >
> > Thanks & Regards,
> > Gaurav Pathak
>

Re: Spark on Windows platform

Posted by Jörn Franke <jo...@gmail.com>.

I think Hortonworks has a Windows Spark distribution. Maybe Bigtop as well? 

> On 29 Feb 2016, at 14:27, gaurav pathak <ga...@gmail.com> wrote:
> 
> Can someone guide me the steps and information regarding, installation of SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be great to read your experiences in using SPARK on Windows platform.
> 
> 
> Thanks & Regards,
> Gaurav Pathak

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark on Windows platform

Posted by Sabarish Sasidharan <sa...@manthan.com>.

If all you want is Spark standalone then its as simple as installing the
binaries and calling Spark submit passing your main class. I would advise
against running on Hadoop on Windows, it's a bit of trouble. But yes you
can do it if you want to.

Regards
Sab

Regards
Sab
On 29-Feb-2016 6:58 pm, "gaurav pathak" <ga...@gmail.com> wrote:

> Can someone guide me the steps and information regarding, installation of
> SPARK on Windows 7/8.1/10 , as well as on Windows Server. Also, it will be
> great to read your experiences in using SPARK on Windows platform.
>
>
> Thanks & Regards,
> Gaurav Pathak
>