Optimal Filesystem (and Settings) for HDFS

[Bob Schulze <b....@ecircle.com>]

We are currently rebuilding our cluster - has anybody recommendations on
the underlaying file system? Just standard Ext3?

I could imagine that the block size could be larger than its default...

Thx for any tips,

	Bob
	

Re: Optimal Filesystem (and Settings) for HDFS

[Allen Wittenauer <aw...@yahoo-inc.com>]

On 5/18/09 11:33 AM, "Edward Capriolo" <ed...@gmail.com> wrote:

> Do not forget 'tune2fs -m 2'. By default this value gets set at 5%.
> With 1 TB disks we got 33 GB more usable space. Talk about instant
> savings!

Yup. Although, I think we're using -m 1.


> On Mon, May 18, 2009 at 1:31 PM, Alex Loddengaard <al...@cloudera.com> wrote:
>> I believe Yahoo! uses ext3,

Yup.  They won't buy me enough battery backed RAM to use a memory file
system. ;)

Re: Optimal Filesystem (and Settings) for HDFS

[Edward Capriolo <ed...@gmail.com>]

Do not forget 'tune2fs -m 2'. By default this value gets set at 5%.
With 1 TB disks we got 33 GB more usable space. Talk about instant
savings!

On Mon, May 18, 2009 at 1:31 PM, Alex Loddengaard <al...@cloudera.com> wrote:
> I believe Yahoo! uses ext3, though I know other people have said that XFS
> has performed better in various benchmarks.  We use ext3, though we haven't
> done any benchmarks to prove its worth.
>
> This question has come up a lot, so I think it'd be worth doing a benchmark
> and writing up the results.  I haven't been able to find a detailed analysis
> / benchmark writeup comparing various filesystems, unfortunately.
>
> Hope this helps,
>
> Alex
>
> On Mon, May 18, 2009 at 8:54 AM, Bob Schulze <b....@ecircle.com> wrote:
>
>> We are currently rebuilding our cluster - has anybody recommendations on
>> the underlaying file system? Just standard Ext3?
>>
>> I could imagine that the block size could be larger than its default...
>>
>> Thx for any tips,
>>
>>        Bob
>>
>>
>

Re: Optimal Filesystem (and Settings) for HDFS

[Steve Loughran <st...@apache.org>]

Bryan Duxbury wrote:
> We use XFS for our data drives, and we've had somewhat mixed results. 


Thanks for that. I've just created a wiki page to put some of these 
notes up -extensions and some hard data would be welcome

http://wiki.apache.org/hadoop/DiskSetup

One problem we have for hard data is that we need some different 
benchmarks for MR jobs. Terasort is good for measuring IO and MR 
framework performance, but for more CPU intensive algorithms, or things 
that need to seek round a bit more, you can't be sure that terasort 
benchmarks are a good predictor of what's right for you in terms of 
hardware, filesystem, etc.

Contributions in this area would be welcome.

I'd like to measure the power consumed on a run too, which is actually 
possible as far as my laptop is concerned, because you can ask it's 
battery what happened.

-steve

Re: Optimal Filesystem (and Settings) for HDFS

[Bryan Duxbury <br...@rapleaf.com>]

We use XFS for our data drives, and we've had somewhat mixed results.  
One of the biggest pros is that XFS has more free space than ext3,  
even with the reserved space settings turned all the way to 0.  
Another is that you can format a 1TB drive as XFS in about 0 seconds,  
versus minutes for ext3. This makes it really fast to kickstart our  
worker nodes.

We have seen some weird stuff happen though when machines run out of  
memory, apparently because the XFS driver does something odd with  
kernel memory. When this happens, we end up having to do some fscking  
before we can get that node back online.

As far as outright performance, I actually *did* do some tests of xfs  
vs ext3 performance on our cluster. If you just look at a single  
machine's local disk speed, you can write and read noticeably faster  
when using XFS instead of ext3. However, the reality is that this  
extra disk performance won't have much of an effect on your overall  
job completion performance, since you will find yourself network  
bottlenecked well in advance of even ext3's performance.

The long and short of it is that we use XFS to speed up our new  
machine deployment, and that's it.

-Bryan

On May 18, 2009, at 10:31 AM, Alex Loddengaard wrote:

> I believe Yahoo! uses ext3, though I know other people have said  
> that XFS
> has performed better in various benchmarks.  We use ext3, though we  
> haven't
> done any benchmarks to prove its worth.
>
> This question has come up a lot, so I think it'd be worth doing a  
> benchmark
> and writing up the results.  I haven't been able to find a detailed  
> analysis
> / benchmark writeup comparing various filesystems, unfortunately.
>
> Hope this helps,
>
> Alex
>
> On Mon, May 18, 2009 at 8:54 AM, Bob Schulze  
> <b....@ecircle.com> wrote:
>
>> We are currently rebuilding our cluster - has anybody  
>> recommendations on
>> the underlaying file system? Just standard Ext3?
>>
>> I could imagine that the block size could be larger than its  
>> default...
>>
>> Thx for any tips,
>>
>>        Bob
>>
>>

Re: Optimal Filesystem (and Settings) for HDFS

[Alex Loddengaard <al...@cloudera.com>]

I believe Yahoo! uses ext3, though I know other people have said that XFS
has performed better in various benchmarks.  We use ext3, though we haven't
done any benchmarks to prove its worth.

This question has come up a lot, so I think it'd be worth doing a benchmark
and writing up the results.  I haven't been able to find a detailed analysis
/ benchmark writeup comparing various filesystems, unfortunately.

Hope this helps,

Alex

On Mon, May 18, 2009 at 8:54 AM, Bob Schulze <b....@ecircle.com> wrote:

> We are currently rebuilding our cluster - has anybody recommendations on
> the underlaying file system? Just standard Ext3?
>
> I could imagine that the block size could be larger than its default...
>
> Thx for any tips,
>
>        Bob
>
>