You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Manickam P <ma...@outlook.com> on 2013/09/17 17:03:47 UTC
Yarn and HDFS federation
Hello Experts,
I know that hdfs federation helps adding multiple name nodes. In this case how this Yarn works? Where will be the Resource Manger if i have multiple name nodes? (Normally in Master Node I guess) Can we execute yarn applications in hadoop cluster without having this hdfs federation?
Please clarify my doubt.
Thanks,
Manickam p
Re: Yarn and HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Using Federated Namespaces is optional.
YARN's services use HDFS minimally on their own. It is your MR job or
your YARN application that really uses the filesystem. So you can have
YARN host itself on any namespace, and it wouldn't matter to your
actual jobs/apps, which will submit with their own FS configured.
(There may be some caveats that apply with use of JHS and where the MR
staging dir goes, but it can be fixed via consistent configs)
On Tue, Sep 17, 2013 at 8:33 PM, Manickam P <ma...@outlook.com> wrote:
> Hello Experts,
>
> I know that hdfs federation helps adding multiple name nodes. In this case
> how this Yarn works?
> Where will be the Resource Manger if i have multiple name nodes? (Normally
> in Master Node I guess)
> Can we execute yarn applications in hadoop cluster without having this hdfs
> federation?
>
> Please clarify my doubt.
>
>
>
> Thanks,
> Manickam p
--
Harsh J
Re: Yarn and HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Using Federated Namespaces is optional.
YARN's services use HDFS minimally on their own. It is your MR job or
your YARN application that really uses the filesystem. So you can have
YARN host itself on any namespace, and it wouldn't matter to your
actual jobs/apps, which will submit with their own FS configured.
(There may be some caveats that apply with use of JHS and where the MR
staging dir goes, but it can be fixed via consistent configs)
On Tue, Sep 17, 2013 at 8:33 PM, Manickam P <ma...@outlook.com> wrote:
> Hello Experts,
>
> I know that hdfs federation helps adding multiple name nodes. In this case
> how this Yarn works?
> Where will be the Resource Manger if i have multiple name nodes? (Normally
> in Master Node I guess)
> Can we execute yarn applications in hadoop cluster without having this hdfs
> federation?
>
> Please clarify my doubt.
>
>
>
> Thanks,
> Manickam p
--
Harsh J
Re: Yarn and HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Using Federated Namespaces is optional.
YARN's services use HDFS minimally on their own. It is your MR job or
your YARN application that really uses the filesystem. So you can have
YARN host itself on any namespace, and it wouldn't matter to your
actual jobs/apps, which will submit with their own FS configured.
(There may be some caveats that apply with use of JHS and where the MR
staging dir goes, but it can be fixed via consistent configs)
On Tue, Sep 17, 2013 at 8:33 PM, Manickam P <ma...@outlook.com> wrote:
> Hello Experts,
>
> I know that hdfs federation helps adding multiple name nodes. In this case
> how this Yarn works?
> Where will be the Resource Manger if i have multiple name nodes? (Normally
> in Master Node I guess)
> Can we execute yarn applications in hadoop cluster without having this hdfs
> federation?
>
> Please clarify my doubt.
>
>
>
> Thanks,
> Manickam p
--
Harsh J
Re: Yarn and HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Using Federated Namespaces is optional.
YARN's services use HDFS minimally on their own. It is your MR job or
your YARN application that really uses the filesystem. So you can have
YARN host itself on any namespace, and it wouldn't matter to your
actual jobs/apps, which will submit with their own FS configured.
(There may be some caveats that apply with use of JHS and where the MR
staging dir goes, but it can be fixed via consistent configs)
On Tue, Sep 17, 2013 at 8:33 PM, Manickam P <ma...@outlook.com> wrote:
> Hello Experts,
>
> I know that hdfs federation helps adding multiple name nodes. In this case
> how this Yarn works?
> Where will be the Resource Manger if i have multiple name nodes? (Normally
> in Master Node I guess)
> Can we execute yarn applications in hadoop cluster without having this hdfs
> federation?
>
> Please clarify my doubt.
>
>
>
> Thanks,
> Manickam p
--
Harsh J