hdfs fuse mount and namenode out of memory conditions.

[Jason Venner <ja...@attributor.com>]

We recently setup a fuse mount using the 18.2 fuse code, against our 
18.1 hdfs, which has been running stably for some time.

We have about 20 datanodes and 50TB or so in our hdfs. The namenode is 
running an i686 kernel and has been running with -Xmx1500m.
We have 1,492,093 files in our hdfs at present.

Since we put up the fuse mount which reads 2 large files at a time, 
about 6 a second or so, we have had the namenode OOM multiple times.

We have upped the memory foot print to the max allowed for a 32bit jvm.

What memory size should we be running?
Is the fuse mount doing something that is increasing the memory use of 
the namenode?

Happy week all.

Re: hdfs fuse mount and namenode out of memory conditions.

[Pete Wyckoff <pw...@facebook.com>]

Hi Jason,

There's nothing that fuse does that should cause this I don't think. If this is a cat operation, fuse will do this in chunks of 32 K which you can change by mounting with -ordbuffer_size=#bytes.

Do you have the stack trace or the NN log while this app is running? Were you doing the equivalent of this app with dfs -cat or something before with no problems?  1.5 M files sounds like it might be close to the limit of a 1.5GB NN?

-- pete


On 11/17/08 9:29 AM, "Jason Venner" <ja...@attributor.com> wrote:

We recently setup a fuse mount using the 18.2 fuse code, against our
18.1 hdfs, which has been running stably for some time.

We have about 20 datanodes and 50TB or so in our hdfs. The namenode is
running an i686 kernel and has been running with -Xmx1500m.
We have 1,492,093 files in our hdfs at present.

Since we put up the fuse mount which reads 2 large files at a time,
about 6 a second or so, we have had the namenode OOM multiple times.

We have upped the memory foot print to the max allowed for a 32bit jvm.

What memory size should we be running?
Is the fuse mount doing something that is increasing the memory use of
the namenode?

Happy week all.