You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stas Oskin <st...@gmail.com> on 2009/09/13 12:01:19 UTC
Multiple disks for DFS
Hi.
When I specify multiple disks for DFS, does Hadoop distributes the
concurrent writings over the multiple disks?
I mean, to prevent an utilization of a single disk?
Thanks for any info on subject.
Re: Multiple disks for DFS
Posted by Stas Oskin <st...@gmail.com>.
Hi Todd.
Thanks, that's exactly what I looked for.
Regards.
2009/9/15 Todd Lipcon <to...@cloudera.com>
> Hi Stas,
>
> Look at the LocalDirAllocator class. It's not really meant for public
> consumption, so its API might break in future releases, but that's the
> class
> that the hadoop daemons use for round robin behavior.
>
> -Todd
>
> On Tue, Sep 15, 2009 at 4:51 AM, Stas Oskin <st...@gmail.com> wrote:
>
> > Hi.
> >
> > Thanks for the advice, but I actually asked, if I can use the DFS local
> > files (where the file is written locally, and not to DFS), with the
> > round-robin mechanism, for my own needs.
> >
> > Regards.
> >
> > 2009/9/14 Dali Kilani <da...@gmail.com>
> >
> > > Just specify multiple directories (where the different local partitions
> > are
> > > mounted) for dfs.data.dir (hdfs data) and mapred.local.dir
> (intermediate
> > > data) in hdfs-site.xml.Data should then be striped across the different
> > > partitions/disks.
> > >
> > > See here : http://bit.ly/fbUkr
> > >
> > > Dali
> > >
> > > On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <st...@gmail.com>
> > wrote:
> > >
> > > > Hi.
> > > >
> > > > Thanks for the explanation.
> > > >
> > > > Any idea if I can re-use this round robin mechanism for local disk
> > > writing?
> > > >
> > > > Or it's DFS only?
> > > >
> > > > Regards.
> > > >
> > > > 2009/9/14 Jason Venner <ja...@gmail.com>
> > > >
> > > > > When you have multiple partitions specified for hdfs storage, they
> > are
> > > > used
> > > > > for block storage in a round robin fashion.
> > > > > If a partition has insufficient space it is dropped for the set
> used
> > > for
> > > > > storing new blocks.
> > > > >
> > > > > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi.
> > > > > >
> > > > > > When I specify multiple disks for DFS, does Hadoop distributes
> the
> > > > > > concurrent writings over the multiple disks?
> > > > > >
> > > > > > I mean, to prevent an utilization of a single disk?
> > > > > >
> > > > > > Thanks for any info on subject.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > > > > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > > > > www.prohadoopbook.com a community for Hadoop Professionals
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Dali Kilani
> > > ===========
> > > Phone : (650) 492-5921 (Google Voice)
> > > E-Fax : (775) 552-2982
> > >
> >
>
Re: Multiple disks for DFS
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Stas,
Look at the LocalDirAllocator class. It's not really meant for public
consumption, so its API might break in future releases, but that's the class
that the hadoop daemons use for round robin behavior.
-Todd
On Tue, Sep 15, 2009 at 4:51 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Thanks for the advice, but I actually asked, if I can use the DFS local
> files (where the file is written locally, and not to DFS), with the
> round-robin mechanism, for my own needs.
>
> Regards.
>
> 2009/9/14 Dali Kilani <da...@gmail.com>
>
> > Just specify multiple directories (where the different local partitions
> are
> > mounted) for dfs.data.dir (hdfs data) and mapred.local.dir (intermediate
> > data) in hdfs-site.xml.Data should then be striped across the different
> > partitions/disks.
> >
> > See here : http://bit.ly/fbUkr
> >
> > Dali
> >
> > On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <st...@gmail.com>
> wrote:
> >
> > > Hi.
> > >
> > > Thanks for the explanation.
> > >
> > > Any idea if I can re-use this round robin mechanism for local disk
> > writing?
> > >
> > > Or it's DFS only?
> > >
> > > Regards.
> > >
> > > 2009/9/14 Jason Venner <ja...@gmail.com>
> > >
> > > > When you have multiple partitions specified for hdfs storage, they
> are
> > > used
> > > > for block storage in a round robin fashion.
> > > > If a partition has insufficient space it is dropped for the set used
> > for
> > > > storing new blocks.
> > > >
> > > > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi.
> > > > >
> > > > > When I specify multiple disks for DFS, does Hadoop distributes the
> > > > > concurrent writings over the multiple disks?
> > > > >
> > > > > I mean, to prevent an utilization of a single disk?
> > > > >
> > > > > Thanks for any info on subject.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > > > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > > > www.prohadoopbook.com a community for Hadoop Professionals
> > > >
> > >
> >
> >
> >
> > --
> > Dali Kilani
> > ===========
> > Phone : (650) 492-5921 (Google Voice)
> > E-Fax : (775) 552-2982
> >
>
Re: Multiple disks for DFS
Posted by Ted Dunning <te...@gmail.com>.
I think that the question confused people. This is open-source, after all,
so you can use the code any way you like. It sounds like you are asking
permission to use the code which you, of course, already have.
On Tue, Sep 15, 2009 at 4:51 AM, Stas Oskin <st...@gmail.com> wrote:
> ... I actually asked, if I can use the DFS local
> files (where the file is written locally, and not to DFS), with the
> round-robin mechanism, for my own needs.
>
--
Ted Dunning, CTO
DeepDyve
Re: Multiple disks for DFS
Posted by Stas Oskin <st...@gmail.com>.
Hi.
Thanks for the advice, but I actually asked, if I can use the DFS local
files (where the file is written locally, and not to DFS), with the
round-robin mechanism, for my own needs.
Regards.
2009/9/14 Dali Kilani <da...@gmail.com>
> Just specify multiple directories (where the different local partitions are
> mounted) for dfs.data.dir (hdfs data) and mapred.local.dir (intermediate
> data) in hdfs-site.xml.Data should then be striped across the different
> partitions/disks.
>
> See here : http://bit.ly/fbUkr
>
> Dali
>
> On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <st...@gmail.com> wrote:
>
> > Hi.
> >
> > Thanks for the explanation.
> >
> > Any idea if I can re-use this round robin mechanism for local disk
> writing?
> >
> > Or it's DFS only?
> >
> > Regards.
> >
> > 2009/9/14 Jason Venner <ja...@gmail.com>
> >
> > > When you have multiple partitions specified for hdfs storage, they are
> > used
> > > for block storage in a round robin fashion.
> > > If a partition has insufficient space it is dropped for the set used
> for
> > > storing new blocks.
> > >
> > > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com>
> > wrote:
> > >
> > > > Hi.
> > > >
> > > > When I specify multiple disks for DFS, does Hadoop distributes the
> > > > concurrent writings over the multiple disks?
> > > >
> > > > I mean, to prevent an utilization of a single disk?
> > > >
> > > > Thanks for any info on subject.
> > > >
> > >
> > >
> > >
> > > --
> > > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > > www.prohadoopbook.com a community for Hadoop Professionals
> > >
> >
>
>
>
> --
> Dali Kilani
> ===========
> Phone : (650) 492-5921 (Google Voice)
> E-Fax : (775) 552-2982
>
Re: Multiple disks for DFS
Posted by Dali Kilani <da...@gmail.com>.
Just specify multiple directories (where the different local partitions are
mounted) for dfs.data.dir (hdfs data) and mapred.local.dir (intermediate
data) in hdfs-site.xml.Data should then be striped across the different
partitions/disks.
See here : http://bit.ly/fbUkr
Dali
On Mon, Sep 14, 2009 at 11:50 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Thanks for the explanation.
>
> Any idea if I can re-use this round robin mechanism for local disk writing?
>
> Or it's DFS only?
>
> Regards.
>
> 2009/9/14 Jason Venner <ja...@gmail.com>
>
> > When you have multiple partitions specified for hdfs storage, they are
> used
> > for block storage in a round robin fashion.
> > If a partition has insufficient space it is dropped for the set used for
> > storing new blocks.
> >
> > On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com>
> wrote:
> >
> > > Hi.
> > >
> > > When I specify multiple disks for DFS, does Hadoop distributes the
> > > concurrent writings over the multiple disks?
> > >
> > > I mean, to prevent an utilization of a single disk?
> > >
> > > Thanks for any info on subject.
> > >
> >
> >
> >
> > --
> > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > www.prohadoopbook.com a community for Hadoop Professionals
> >
>
--
Dali Kilani
===========
Phone : (650) 492-5921 (Google Voice)
E-Fax : (775) 552-2982
Re: Multiple disks for DFS
Posted by Stas Oskin <st...@gmail.com>.
Hi.
Thanks for the explanation.
Any idea if I can re-use this round robin mechanism for local disk writing?
Or it's DFS only?
Regards.
2009/9/14 Jason Venner <ja...@gmail.com>
> When you have multiple partitions specified for hdfs storage, they are used
> for block storage in a round robin fashion.
> If a partition has insufficient space it is dropped for the set used for
> storing new blocks.
>
> On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com> wrote:
>
> > Hi.
> >
> > When I specify multiple disks for DFS, does Hadoop distributes the
> > concurrent writings over the multiple disks?
> >
> > I mean, to prevent an utilization of a single disk?
> >
> > Thanks for any info on subject.
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Multiple disks for DFS
Posted by Jason Venner <ja...@gmail.com>.
When you have multiple partitions specified for hdfs storage, they are used
for block storage in a round robin fashion.
If a partition has insufficient space it is dropped for the set used for
storing new blocks.
On Sun, Sep 13, 2009 at 3:01 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> When I specify multiple disks for DFS, does Hadoop distributes the
> concurrent writings over the multiple disks?
>
> I mean, to prevent an utilization of a single disk?
>
> Thanks for any info on subject.
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals