Re: du reserved can ran into problems with reserved disk capacity by tune2fs

[Alexander Fahlke <al...@googlemail.com>]

Hi!

At least I'm not alone with this issue. I'd like to create this ticket as I
incidently ran into this again today with a few nodes. :(

On which hadoop-version did you ran into this issue? I guess its not
version related.
It will enhance the ticket if it would not only affect the old hadoop
version I used. ;)

Cheers



On Sun, Dec 1, 2013 at 9:24 PM, Adam Kawa <ka...@gmail.com> wrote:

> We ran into issue as well on our cluster.
>
> +1 for JIRA for that
> Alexander, could you please create a JIRA in
> https://issues.apache.org/jira/browse/HDFS for that (it is your
> observation, so that you should get credit ;). Otherwise, I can do that.
>
>
> 2013/2/12 Alexander Fahlke <al...@googlemail.com>
>
> > Hi!
> >
> > I'm using hadoop-0.20.2 on Debian Squeeze and ran into the same confusion
> > as many others with the parameter for dfs.datanode.du.reserved.
> > One day some data nodes got out of disk errors although there was space
> > left on the disks.
> >
> > The following values are rounded to make the problem more clear:
> >
> > - the disk for the DFS data has 1000GB and only one Partition (ext3) for
> > DFS data
> > - you plan to set the dfs.datanode.du.reserved to 20GB
> > - the reserved reserved-blocks-percentage by tune2fs is 5% (the default)
> >
> > That gives all users, except root, 5% less capacity that they can use.
> > Although the System reports the total of 1000GB as usable for all users
> via
> > df.
> > The hadoop-deamons are not running as root.
> >
> >
> > If i read it right, than hadoop get's the free capacity via df.
> >
> > Starting in
> /src/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
> > on line 350:
> >  return usage.getCapacity()-reserved;
> >
> > going to /src/core/org/apache/hadoop/fs/DF.java which says:
> >  "Filesystem disk space usage statistics. Uses the unix 'df' program"
> >
> >
> > When you have 5% reserved by tune2fs (in our case 50GB) and you give
> > dfs.datanode.du.reserved only 20GB, than you can possibly ran into out of
> > disk errors that hadoop can't handle.
> >
> > In this case you must add the planned 20GB du reserved to the reserved
> > capacity by tune2fs. This results in (at least) 70GB
> > for dfs.datanode.du.reserved in my case.
> >
> >
> > Two ideas:
> >
> > 1. The documentation must be clear at this point to avoid this problem.
> > 2. Hadoop could check for reserved space by tune2fs (or other tools) and
> > add this value to the dfs.datanode.du.reserved parameter.
> >
> >
> > --
> > BR
> >
> > Alexander Fahlke
> > Software Development
> > www.fahlke.org
> >
>



-- 
BR

Alexander Fahlke
Software Development
www.nurago.com | www.fahlke.org