You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by My List <my...@gmail.com> on 2016/04/18 13:48:46 UTC
How to start HDFS on Spark Standalone
Hi ,
I am a newbie on Spark.I wanted to know how to start and verify if HDFS has
started on Spark stand alone.
Env -
Windows 7 - 64 bit
Spark 1.4.1 With Hadoop 2.6
Using Scala Shell - spark-shell
--
Thanks,
Harry
Re: How to start HDFS on Spark Standalone
Posted by Steve Loughran <st...@hortonworks.com>.
1. You don't need to start HDFS or anything like that, just set up Spark so that it can use the Hadoop APIs for some things; on windows this depends on some native libs. This means you don't need to worry about learning it. Focus on the Spark APIs, python and/or scala
2. You should be able to find the native windows binaries here: https://github.com/steveloughran/winutils
These are either version I lifted out of HDP for windows, or, more recently, checking out and building on Windows the same git commit as used/voted in for the ASF releases.
3. For performance, you also need the native libs for compression codecs: snappy, LZO &c. I see I've put them in the windows 2.7.1 release, but not the others (ASF mvn package doesn't add them): https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin ...
If the version of the hadoop libs aren't working with one of those versions I've put up, ping me and I'll build up the relevant binaries out of the ASF
From: My List <my...@gmail.com>>
Date: Monday, 18 April 2016 at 13:13
To: Deepak Sharma <de...@gmail.com>>
Cc: SparkUser <us...@spark.apache.org>>
Subject: Re: How to start HDFS on Spark Standalone
Deepak,
The following could be a very dumb questions so pardon me for the same.
1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) does it not come in the zip or tar file?
2) If it does not come along,Is there a Apache Hadoop for windows, is it in binary format or will have to build it?
3) Is there a basic tutorial for Hadoop on windows for the basic needs of Spark.
Thanks in Advance !
On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>> wrote:
Once you download hadoop and format the namenode , you can use start-dfs.sh to start hdfs.
Then use 'jps' to sss if datanode/namenode services are up and running.
Thanks
Deepak
On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com>> wrote:
Hi ,
I am a newbie on Spark.I wanted to know how to start and verify if HDFS has started on Spark stand alone.
Env -
Windows 7 - 64 bit
Spark 1.4.1 With Hadoop 2.6
Using Scala Shell - spark-shell
--
Thanks,
Harry
--
Thanks
Deepak
www.bigdatabig.com<http://www.bigdatabig.com>
www.keosha.net<http://www.keosha.net>
--
Thanks,
Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Michael Segel <ms...@hotmail.com>.
Perhaps this is a silly question on my part….
Why do you want to start up HDFS on a single node?
You only mention one windows machine in your description of your cluster.
If this is a learning experience, why not run Hadoop in a VM (MapR and I think the other vendors make linux images that can run in a VM)
HTH
-Mike
> On Apr 18, 2016, at 10:27 AM, Jörn Franke <jo...@gmail.com> wrote:
>
> I think the easiest would be to use a Hadoop Windows distribution, such as Hortonworks. However, the Linux version of Hortonworks is a little bit more advanced.
>
> On 18 Apr 2016, at 14:13, My List <mylisttech@gmail.com <ma...@gmail.com>> wrote:
>
>> Deepak,
>>
>> The following could be a very dumb questions so pardon me for the same.
>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) does it not come in the zip or tar file?
>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it in binary format or will have to build it?
>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of Spark.
>>
>> Thanks in Advance !
>>
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <deepakmca05@gmail.com <ma...@gmail.com>> wrote:
>> Once you download hadoop and format the namenode , you can use start-dfs.sh to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:18 PM, My List <mylisttech@gmail.com <ma...@gmail.com>> wrote:
>> Hi ,
>>
>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS has started on Spark stand alone.
>>
>> Env -
>> Windows 7 - 64 bit
>> Spark 1.4.1 With Hadoop 2.6
>>
>> Using Scala Shell - spark-shell
>>
>>
>> --
>> Thanks,
>> Harry
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com <http://www.bigdatabig.com/>
>> www.keosha.net <http://www.keosha.net/>
>>
>>
>> --
>> Thanks,
>> Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Jörn Franke <jo...@gmail.com>.
I think the easiest would be to use a Hadoop Windows distribution, such as Hortonworks. However, the Linux version of Hortonworks is a little bit more advanced.
> On 18 Apr 2016, at 14:13, My List <my...@gmail.com> wrote:
>
> Deepak,
>
> The following could be a very dumb questions so pardon me for the same.
> 1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) does it not come in the zip or tar file?
> 2) If it does not come along,Is there a Apache Hadoop for windows, is it in binary format or will have to build it?
> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of Spark.
>
> Thanks in Advance !
>
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com> wrote:
>> Once you download hadoop and format the namenode , you can use start-dfs.sh to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>
>> Thanks
>> Deepak
>>
>>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>>> Hi ,
>>>
>>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS has started on Spark stand alone.
>>>
>>> Env -
>>> Windows 7 - 64 bit
>>> Spark 1.4.1 With Hadoop 2.6
>>>
>>> Using Scala Shell - spark-shell
>>>
>>>
>>> --
>>> Thanks,
>>> Harry
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>
>
>
> --
> Thanks,
> Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Deepak Sharma <de...@gmail.com>.
I prefer using CentOS/SLES/Ubuntu personally.
Thanks
Deepak
On Mon, Apr 18, 2016 at 5:57 PM, My List <my...@gmail.com> wrote:
> Deepak,
>
> I love the unix flavors have been a programmed on them. Just have a
> windows laptop and pc hence haven't moved to unix flavors. Was trying to
> run big data stuff on windows. Have run in so much of issues that I could
> just throw the laptop with windows out.
>
> Your view - Redhat, Ubuntu or Centos.
> Does Redhat give a one year licence on purchase etc?
>
> Thanks
>
> On Mon, Apr 18, 2016 at 5:52 PM, Deepak Sharma <de...@gmail.com>
> wrote:
>
>> It works well with all flavors of Linux.
>> It all depends on your ex with these flavors.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:51 PM, My List <my...@gmail.com> wrote:
>>
>>> Deepak,
>>>
>>> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
>>> and issues are Galore on Spark.
>>> Since I am starting afresh, what would you advice?
>>>
>>> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma <de...@gmail.com>
>>> wrote:
>>>
>>>> Binary for Spark means ts spark built against hadoop 2.6
>>>> It will not have any hadoop executables.
>>>> You'll have to setup hadoop separately.
>>>> I have not used windows version yet but there are some.
>>>>
>>>> Thanks
>>>> Deepak
>>>>
>>>> On Mon, Apr 18, 2016 at 5:43 PM, My List <my...@gmail.com> wrote:
>>>>
>>>>> Deepak,
>>>>>
>>>>> The following could be a very dumb questions so pardon me for the same.
>>>>> 1) When I download the binary for Spark with a version of
>>>>> Hadoop(Hadoop 2.6) does it not come in the zip or tar file?
>>>>> 2) If it does not come along,Is there a Apache Hadoop for windows, is
>>>>> it in binary format or will have to build it?
>>>>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs
>>>>> of Spark.
>>>>>
>>>>> Thanks in Advance !
>>>>>
>>>>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Once you download hadoop and format the namenode , you can use
>>>>>> start-dfs.sh to start hdfs.
>>>>>> Then use 'jps' to sss if datanode/namenode services are up and
>>>>>> running.
>>>>>>
>>>>>> Thanks
>>>>>> Deepak
>>>>>>
>>>>>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi ,
>>>>>>>
>>>>>>> I am a newbie on Spark.I wanted to know how to start and verify if
>>>>>>> HDFS has started on Spark stand alone.
>>>>>>>
>>>>>>> Env -
>>>>>>> Windows 7 - 64 bit
>>>>>>> Spark 1.4.1 With Hadoop 2.6
>>>>>>>
>>>>>>> Using Scala Shell - spark-shell
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Harry
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks
>>>>>> Deepak
>>>>>> www.bigdatabig.com
>>>>>> www.keosha.net
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Harmeet
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Deepak
>>>> www.bigdatabig.com
>>>> www.keosha.net
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Harmeet
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: How to start HDFS on Spark Standalone
Posted by My List <my...@gmail.com>.
Deepak,
I love the unix flavors have been a programmed on them. Just have a windows
laptop and pc hence haven't moved to unix flavors. Was trying to run big
data stuff on windows. Have run in so much of issues that I could just
throw the laptop with windows out.
Your view - Redhat, Ubuntu or Centos.
Does Redhat give a one year licence on purchase etc?
Thanks
On Mon, Apr 18, 2016 at 5:52 PM, Deepak Sharma <de...@gmail.com>
wrote:
> It works well with all flavors of Linux.
> It all depends on your ex with these flavors.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:51 PM, My List <my...@gmail.com> wrote:
>
>> Deepak,
>>
>> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
>> and issues are Galore on Spark.
>> Since I am starting afresh, what would you advice?
>>
>> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma <de...@gmail.com>
>> wrote:
>>
>>> Binary for Spark means ts spark built against hadoop 2.6
>>> It will not have any hadoop executables.
>>> You'll have to setup hadoop separately.
>>> I have not used windows version yet but there are some.
>>>
>>> Thanks
>>> Deepak
>>>
>>> On Mon, Apr 18, 2016 at 5:43 PM, My List <my...@gmail.com> wrote:
>>>
>>>> Deepak,
>>>>
>>>> The following could be a very dumb questions so pardon me for the same.
>>>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
>>>> 2.6) does it not come in the zip or tar file?
>>>> 2) If it does not come along,Is there a Apache Hadoop for windows, is
>>>> it in binary format or will have to build it?
>>>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs
>>>> of Spark.
>>>>
>>>> Thanks in Advance !
>>>>
>>>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
>>>> wrote:
>>>>
>>>>> Once you download hadoop and format the namenode , you can use
>>>>> start-dfs.sh to start hdfs.
>>>>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>>>>
>>>>> Thanks
>>>>> Deepak
>>>>>
>>>>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>>>>>
>>>>>> Hi ,
>>>>>>
>>>>>> I am a newbie on Spark.I wanted to know how to start and verify if
>>>>>> HDFS has started on Spark stand alone.
>>>>>>
>>>>>> Env -
>>>>>> Windows 7 - 64 bit
>>>>>> Spark 1.4.1 With Hadoop 2.6
>>>>>>
>>>>>> Using Scala Shell - spark-shell
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Harry
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks
>>>>> Deepak
>>>>> www.bigdatabig.com
>>>>> www.keosha.net
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Harmeet
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>
>>
>> --
>> Thanks,
>> Harmeet
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
--
Thanks,
Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Deepak Sharma <de...@gmail.com>.
It works well with all flavors of Linux.
It all depends on your ex with these flavors.
Thanks
Deepak
On Mon, Apr 18, 2016 at 5:51 PM, My List <my...@gmail.com> wrote:
> Deepak,
>
> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
> and issues are Galore on Spark.
> Since I am starting afresh, what would you advice?
>
> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma <de...@gmail.com>
> wrote:
>
>> Binary for Spark means ts spark built against hadoop 2.6
>> It will not have any hadoop executables.
>> You'll have to setup hadoop separately.
>> I have not used windows version yet but there are some.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:43 PM, My List <my...@gmail.com> wrote:
>>
>>> Deepak,
>>>
>>> The following could be a very dumb questions so pardon me for the same.
>>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
>>> 2.6) does it not come in the zip or tar file?
>>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
>>> in binary format or will have to build it?
>>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs
>>> of Spark.
>>>
>>> Thanks in Advance !
>>>
>>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
>>> wrote:
>>>
>>>> Once you download hadoop and format the namenode , you can use
>>>> start-dfs.sh to start hdfs.
>>>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>>>
>>>> Thanks
>>>> Deepak
>>>>
>>>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>>>>
>>>>> Hi ,
>>>>>
>>>>> I am a newbie on Spark.I wanted to know how to start and verify if
>>>>> HDFS has started on Spark stand alone.
>>>>>
>>>>> Env -
>>>>> Windows 7 - 64 bit
>>>>> Spark 1.4.1 With Hadoop 2.6
>>>>>
>>>>> Using Scala Shell - spark-shell
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Harry
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks
>>>> Deepak
>>>> www.bigdatabig.com
>>>> www.keosha.net
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Harmeet
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: How to start HDFS on Spark Standalone
Posted by My List <my...@gmail.com>.
Deepak,
Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
and issues are Galore on Spark.
Since I am starting afresh, what would you advice?
On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma <de...@gmail.com>
wrote:
> Binary for Spark means ts spark built against hadoop 2.6
> It will not have any hadoop executables.
> You'll have to setup hadoop separately.
> I have not used windows version yet but there are some.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:43 PM, My List <my...@gmail.com> wrote:
>
>> Deepak,
>>
>> The following could be a very dumb questions so pardon me for the same.
>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
>> 2.6) does it not come in the zip or tar file?
>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
>> in binary format or will have to build it?
>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of
>> Spark.
>>
>> Thanks in Advance !
>>
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
>> wrote:
>>
>>> Once you download hadoop and format the namenode , you can use
>>> start-dfs.sh to start hdfs.
>>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>>
>>> Thanks
>>> Deepak
>>>
>>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>>>
>>>> Hi ,
>>>>
>>>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
>>>> has started on Spark stand alone.
>>>>
>>>> Env -
>>>> Windows 7 - 64 bit
>>>> Spark 1.4.1 With Hadoop 2.6
>>>>
>>>> Using Scala Shell - spark-shell
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Harry
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>
>>
>> --
>> Thanks,
>> Harmeet
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
--
Thanks,
Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Deepak Sharma <de...@gmail.com>.
Binary for Spark means ts spark built against hadoop 2.6
It will not have any hadoop executables.
You'll have to setup hadoop separately.
I have not used windows version yet but there are some.
Thanks
Deepak
On Mon, Apr 18, 2016 at 5:43 PM, My List <my...@gmail.com> wrote:
> Deepak,
>
> The following could be a very dumb questions so pardon me for the same.
> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
> 2.6) does it not come in the zip or tar file?
> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
> in binary format or will have to build it?
> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of
> Spark.
>
> Thanks in Advance !
>
> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
> wrote:
>
>> Once you download hadoop and format the namenode , you can use
>> start-dfs.sh to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>>
>>> Hi ,
>>>
>>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
>>> has started on Spark stand alone.
>>>
>>> Env -
>>> Windows 7 - 64 bit
>>> Spark 1.4.1 With Hadoop 2.6
>>>
>>> Using Scala Shell - spark-shell
>>>
>>>
>>> --
>>> Thanks,
>>> Harry
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: How to start HDFS on Spark Standalone
Posted by My List <my...@gmail.com>.
Deepak,
The following could be a very dumb questions so pardon me for the same.
1) When I download the binary for Spark with a version of Hadoop(Hadoop
2.6) does it not come in the zip or tar file?
2) If it does not come along,Is there a Apache Hadoop for windows, is it in
binary format or will have to build it?
3) Is there a basic tutorial for Hadoop on windows for the basic needs of
Spark.
Thanks in Advance !
On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma <de...@gmail.com>
wrote:
> Once you download hadoop and format the namenode , you can use
> start-dfs.sh to start hdfs.
> Then use 'jps' to sss if datanode/namenode services are up and running.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
>
>> Hi ,
>>
>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
>> has started on Spark stand alone.
>>
>> Env -
>> Windows 7 - 64 bit
>> Spark 1.4.1 With Hadoop 2.6
>>
>> Using Scala Shell - spark-shell
>>
>>
>> --
>> Thanks,
>> Harry
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
--
Thanks,
Harmeet
Re: How to start HDFS on Spark Standalone
Posted by Deepak Sharma <de...@gmail.com>.
Once you download hadoop and format the namenode , you can use start-dfs.sh
to start hdfs.
Then use 'jps' to sss if datanode/namenode services are up and running.
Thanks
Deepak
On Mon, Apr 18, 2016 at 5:18 PM, My List <my...@gmail.com> wrote:
> Hi ,
>
> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
> has started on Spark stand alone.
>
> Env -
> Windows 7 - 64 bit
> Spark 1.4.1 With Hadoop 2.6
>
> Using Scala Shell - spark-shell
>
>
> --
> Thanks,
> Harry
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net