You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Marco Capuccini <ma...@farmbio.uu.se> on 2016/06/05 09:50:23 UTC
Does Spark uses data locality information from HDFS when running in
standalone mode?
Dear all,
Does Spark uses data locality information from HDFS, when running in standalone mode? Or is it running on YARN mandatory for such purpose? I can't find this information in the docs, and on Google I am only finding contrasting opinion on that.
Regards
Marco Capuccini
Re: Does Spark uses data locality information from HDFS when running
in standalone mode?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Actually it is an interesting question. Spark standalone uses simple
cluster manager that is included with Spark. However, I am not sure that
simple cluster manager can work out the whereabouts of datanodes in Hadoop
cluster. I start YARN with HDFS together so don't have this concern
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 5 June 2016 at 14:09, Mich Talebzadeh <mi...@gmail.com> wrote:
> I use YARN as I run Hive on Spark engine in yarn-cluster mode plus other
> stuff. if I turn off YARN half of my applications won't work. I don't see
> great concern for supporting YARN. However you may have other reasons
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 5 June 2016 at 13:40, Marco Capuccini <ma...@farmbio.uu.se>
> wrote:
>
>> I meant when running in standalone cluster mode, where Hadoop data nodes
>> run on the same nodes where the Spark workers run. I don’t want to support
>> YARN as well in my infrastructure, and since I already set up a standalone
>> Spark cluster, I was wondering if running only HDFS in the same cluster
>> would be enough.
>>
>> Regards
>> Marco
>>
>> On 05 Jun 2016, at 12:17, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>> Well in standalone mode you are running your spark code on one physical
>> node so the assumption would be that there is HDFS node running on the same
>> host.
>>
>> When you are running Spark in yarn-client mode, then Yarn is part of
>> Hadoop core and Yarn will know about the datanodes from
>> %HADOOP_HOME/etc/Hadoop/slaves
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 5 June 2016 at 10:50, Marco Capuccini <ma...@farmbio.uu.se>
>> wrote:
>>
>>> Dear all,
>>>
>>> Does Spark uses data locality information from HDFS, when running in
>>> standalone mode? Or is it running on YARN mandatory for such purpose? I
>>> can't find this information in the docs, and on Google I am only finding
>>> contrasting opinion on that.
>>>
>>> Regards
>>> Marco Capuccini
>>>
>>
>>
>>
>
Re: Does Spark uses data locality information from HDFS when running
in standalone mode?
Posted by Mich Talebzadeh <mi...@gmail.com>.
I use YARN as I run Hive on Spark engine in yarn-cluster mode plus other
stuff. if I turn off YARN half of my applications won't work. I don't see
great concern for supporting YARN. However you may have other reasons
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 5 June 2016 at 13:40, Marco Capuccini <ma...@farmbio.uu.se>
wrote:
> I meant when running in standalone cluster mode, where Hadoop data nodes
> run on the same nodes where the Spark workers run. I don’t want to support
> YARN as well in my infrastructure, and since I already set up a standalone
> Spark cluster, I was wondering if running only HDFS in the same cluster
> would be enough.
>
> Regards
> Marco
>
> On 05 Jun 2016, at 12:17, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Well in standalone mode you are running your spark code on one physical
> node so the assumption would be that there is HDFS node running on the same
> host.
>
> When you are running Spark in yarn-client mode, then Yarn is part of
> Hadoop core and Yarn will know about the datanodes from
> %HADOOP_HOME/etc/Hadoop/slaves
>
> HTH
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 5 June 2016 at 10:50, Marco Capuccini <ma...@farmbio.uu.se>
> wrote:
>
>> Dear all,
>>
>> Does Spark uses data locality information from HDFS, when running in
>> standalone mode? Or is it running on YARN mandatory for such purpose? I
>> can't find this information in the docs, and on Google I am only finding
>> contrasting opinion on that.
>>
>> Regards
>> Marco Capuccini
>>
>
>
>
Re: Does Spark uses data locality information from HDFS when
running in standalone mode?
Posted by Marco Capuccini <ma...@farmbio.uu.se>.
I meant when running in standalone cluster mode, where Hadoop data nodes run on the same nodes where the Spark workers run. I don’t want to support YARN as well in my infrastructure, and since I already set up a standalone Spark cluster, I was wondering if running only HDFS in the same cluster would be enough.
Regards
Marco
On 05 Jun 2016, at 12:17, Mich Talebzadeh <mi...@gmail.com>> wrote:
Well in standalone mode you are running your spark code on one physical node so the assumption would be that there is HDFS node running on the same host.
When you are running Spark in yarn-client mode, then Yarn is part of Hadoop core and Yarn will know about the datanodes from %HADOOP_HOME/etc/Hadoop/slaves
HTH
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 5 June 2016 at 10:50, Marco Capuccini <ma...@farmbio.uu.se>> wrote:
Dear all,
Does Spark uses data locality information from HDFS, when running in standalone mode? Or is it running on YARN mandatory for such purpose? I can't find this information in the docs, and on Google I am only finding contrasting opinion on that.
Regards
Marco Capuccini
Re: Does Spark uses data locality information from HDFS when running
in standalone mode?
Posted by Mich Talebzadeh <mi...@gmail.com>.
Well in standalone mode you are running your spark code on one physical
node so the assumption would be that there is HDFS node running on the same
host.
When you are running Spark in yarn-client mode, then Yarn is part of Hadoop
core and Yarn will know about the datanodes from
%HADOOP_HOME/etc/Hadoop/slaves
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 5 June 2016 at 10:50, Marco Capuccini <ma...@farmbio.uu.se>
wrote:
> Dear all,
>
> Does Spark uses data locality information from HDFS, when running in
> standalone mode? Or is it running on YARN mandatory for such purpose? I
> can't find this information in the docs, and on Google I am only finding
> contrasting opinion on that.
>
> Regards
> Marco Capuccini
>
Re: Does Spark uses data locality information from HDFS when running
in standalone mode?
Posted by Eugene Morozov <ev...@gmail.com>.
Marco,
I'd say yes, because it uses different implementation of hadoop's
InputFormat interface underneath.
What kind of proof would you like to see?
--
Be well!
Jean Morozov
On Sun, Jun 5, 2016 at 12:50 PM, Marco Capuccini <
marco.capuccini@farmbio.uu.se> wrote:
> Dear all,
>
> Does Spark uses data locality information from HDFS, when running in
> standalone mode? Or is it running on YARN mandatory for such purpose? I
> can't find this information in the docs, and on Google I am only finding
> contrasting opinion on that.
>
> Regards
> Marco Capuccini
>