You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "David C. Kerber" <dk...@warrenrogersassociates.com> on 2008/10/24 18:41:42 UTC
Using hadoop as storage cluster?
Hi -
I'm a complete newbie to hadoop, and am wondering if it's appropriate for configuring a bunch of older machines that have no other use, for use as a storage cluster on an otherwise windows network, so that my windows clients see their combined disk space as a single large share?
If so, will I need additional software to let the windows clients see them (like Samba does for a single machine)? We don't have a lot of linux experience in our office, but probably enough to get this going if it's not too complex; mostly with Ubuntu and Fedora.
If hadoop isn't well-suited to this use, or there is something better, I'm open to suggestions....
Thanks
----------------------------------
David Kerber
Warren Rogers Associates
(800)-972-7472 x-111
dkerber@WarrenRogersAssociates.com
www.WarrenRogersAssociates.com
----------------------------------
Re: Using hadoop as storage cluster?
Posted by Alex Loddengaard <al...@cloudera.com>.
I don't think HDFS would be the ideal DFS for you. The amount of meta data
associated with even small files is large, so you would most likely bog down
your namenode. HDFS is meant for large files. Take a look at Wikipedia for
more DFS implementations:
<http://en.wikipedia.org/wiki/List_of_file_systems#Distributed_file_systems>
I don't know much about other DFS options, though, so maybe someone else
could say more?
Alex
On Fri, Oct 24, 2008 at 5:42 PM, David C. Kerber <
dkerber@warrenrogersassociates.com> wrote:
> Are there any tuning settings that can be adjusted to optimize for files of
> a given size range?
>
> There would be quite a few files in the 100kB to 2MB range, which are
> received and processed daily, with smaller numbers ranging up to ~600MB or
> so which are summarizations of many of the daily data files, and maybe a
> handful in the 1GB - 6GB range (disk images and database backups, mostly).
> There would also be a few (comparatively few, that is) configuration files
> of a few kB each.
>
> Thanks for the response; do you know of any other systems with similar
> functionality?
>
> Dave
>
>
>
> > -----Original Message-----
> > From: Alex Loddengaard [mailto:alex@cloudera.com]
> > Sent: Friday, October 24, 2008 5:42 PM
> > To: core-user@hadoop.apache.org
> > Subject: Re: Using hadoop as storage cluster?
> >
> > What files do you expect to be storing? Generally speaking,
> > HDFS (Hadoop's distributed file system) does not handle small
> > files very efficiently.
> > Instead it's optimized for large files, upwards of 64MB each.
> >
> > Alex
> >
> > On Fri, Oct 24, 2008 at 9:41 AM, David C. Kerber <
> > dkerber@warrenrogersassociates.com> wrote:
> >
> > > Hi -
> > >
> > > I'm a complete newbie to hadoop, and am wondering if it's
> > appropriate
> > > for configuring a bunch of older machines that have no
> > other use, for
> > > use as a storage cluster on an otherwise windows network,
> > so that my
> > > windows clients see their combined disk space as a single
> > large share?
> > >
> > > If so, will I need additional software to let the windows
> > clients see
> > > them (like Samba does for a single machine)? We don't have
> > a lot of
> > > linux experience in our office, but probably enough to get
> > this going
> > > if it's not too complex; mostly with Ubuntu and Fedora.
> > >
> > > If hadoop isn't well-suited to this use, or there is
> > something better,
> > > I'm open to suggestions....
> > >
> > > Thanks
> > > ----------------------------------
> > > David Kerber
> > > Warren Rogers Associates
> > > (800)-972-7472 x-111
> > > dkerber@WarrenRogersAssociates.com
> > > www.WarrenRogersAssociates.com
> > > ----------------------------------
> > >
> >
>
RE: Using hadoop as storage cluster?
Posted by Malcolm Matalka <mm...@millennialmedia.com>.
You might look into MogileFS if that is the case.
-----Original Message-----
From: Doug Cutting [mailto:cutting@gmail.com] On Behalf Of Doug Cutting
Sent: Monday, October 27, 2008 13:46
To: core-user@hadoop.apache.org
Subject: Re: Using hadoop as storage cluster?
David C. Kerber wrote:
> There would be quite a few files in the 100kB to 2MB range, which are
received and processed daily, with smaller numbers ranging up to ~600MB
or so which are summarizations of many of the daily data files, and
maybe a handful in the 1GB - 6GB range (disk images and database
backups, mostly). There would also be a few (comparatively few, that
is) configuration files of a few kB each.
File size isn't so much the issue as is the total number of files. If
the total number of files that will be in the filesystem at a time is
less than a few million, then you'll probably be fine. If you need ten
million files then you may still be fine if your namenode has a lot of
memory (e.g., 16GB). If you need a lot more than that, then HDFS is
probably not be well suited to your task.
Doug
Re: Using hadoop as storage cluster?
Posted by Doug Cutting <cu...@apache.org>.
David C. Kerber wrote:
> There would be quite a few files in the 100kB to 2MB range, which are received and processed daily, with smaller numbers ranging up to ~600MB or so which are summarizations of many of the daily data files, and maybe a handful in the 1GB - 6GB range (disk images and database backups, mostly). There would also be a few (comparatively few, that is) configuration files of a few kB each.
File size isn't so much the issue as is the total number of files. If
the total number of files that will be in the filesystem at a time is
less than a few million, then you'll probably be fine. If you need ten
million files then you may still be fine if your namenode has a lot of
memory (e.g., 16GB). If you need a lot more than that, then HDFS is
probably not be well suited to your task.
Doug
RE: Using hadoop as storage cluster?
Posted by "David C. Kerber" <dk...@warrenrogersassociates.com>.
Are there any tuning settings that can be adjusted to optimize for files of a given size range?
There would be quite a few files in the 100kB to 2MB range, which are received and processed daily, with smaller numbers ranging up to ~600MB or so which are summarizations of many of the daily data files, and maybe a handful in the 1GB - 6GB range (disk images and database backups, mostly). There would also be a few (comparatively few, that is) configuration files of a few kB each.
Thanks for the response; do you know of any other systems with similar functionality?
Dave
> -----Original Message-----
> From: Alex Loddengaard [mailto:alex@cloudera.com]
> Sent: Friday, October 24, 2008 5:42 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Using hadoop as storage cluster?
>
> What files do you expect to be storing? Generally speaking,
> HDFS (Hadoop's distributed file system) does not handle small
> files very efficiently.
> Instead it's optimized for large files, upwards of 64MB each.
>
> Alex
>
> On Fri, Oct 24, 2008 at 9:41 AM, David C. Kerber <
> dkerber@warrenrogersassociates.com> wrote:
>
> > Hi -
> >
> > I'm a complete newbie to hadoop, and am wondering if it's
> appropriate
> > for configuring a bunch of older machines that have no
> other use, for
> > use as a storage cluster on an otherwise windows network,
> so that my
> > windows clients see their combined disk space as a single
> large share?
> >
> > If so, will I need additional software to let the windows
> clients see
> > them (like Samba does for a single machine)? We don't have
> a lot of
> > linux experience in our office, but probably enough to get
> this going
> > if it's not too complex; mostly with Ubuntu and Fedora.
> >
> > If hadoop isn't well-suited to this use, or there is
> something better,
> > I'm open to suggestions....
> >
> > Thanks
> > ----------------------------------
> > David Kerber
> > Warren Rogers Associates
> > (800)-972-7472 x-111
> > dkerber@WarrenRogersAssociates.com
> > www.WarrenRogersAssociates.com
> > ----------------------------------
> >
>
Re: Using hadoop as storage cluster?
Posted by Alex Loddengaard <al...@cloudera.com>.
What files do you expect to be storing? Generally speaking, HDFS (Hadoop's
distributed file system) does not handle small files very efficiently.
Instead it's optimized for large files, upwards of 64MB each.
Alex
On Fri, Oct 24, 2008 at 9:41 AM, David C. Kerber <
dkerber@warrenrogersassociates.com> wrote:
> Hi -
>
> I'm a complete newbie to hadoop, and am wondering if it's appropriate for
> configuring a bunch of older machines that have no other use, for use as a
> storage cluster on an otherwise windows network, so that my windows clients
> see their combined disk space as a single large share?
>
> If so, will I need additional software to let the windows clients see them
> (like Samba does for a single machine)? We don't have a lot of linux
> experience in our office, but probably enough to get this going if it's not
> too complex; mostly with Ubuntu and Fedora.
>
> If hadoop isn't well-suited to this use, or there is something better, I'm
> open to suggestions....
>
> Thanks
> ----------------------------------
> David Kerber
> Warren Rogers Associates
> (800)-972-7472 x-111
> dkerber@WarrenRogersAssociates.com
> www.WarrenRogersAssociates.com
> ----------------------------------
>