Inconsistence between the datanode volume info and OS df

[Xie Gang <xi...@gmail.com>]

Hi,

Recently, we hit an issue that, there is a difference between the freeSpace
of the datanode volume info and the OS df:

For example:
the jmx of the dn shows:

 "VolumeInfo" :
"{\"/....\":{\"freeSpace\":1445398864500,\"usedSpace\":228138206927,\"reservedSpace\":53687091200}}",

But the df shows:
/dev/sda 2146676656 253778008 1785508084 13% /...

There is about 400GB gap which is regarded as Non DFS used. And the most
strange thing is that, after I restart the dn process, the gap disappear.
And after some days, the gap shows again.

The yarn shared the same server of the dn and has some file cache. Could it
be related?

The direct cause is that the freeSpace from dn is quit different from the
available space from df. After tracking down the code, freeSpace of the dn
is from dirFile.getUsableSpace(). could it have some problem? Do we hit
this issue before?

Thanks,
Gang


-- 
Xie Gang

Re: Inconsistence between the datanode volume info and OS df

[Xie Gang <xi...@gmail.com>]

Got the root cause, it's a dup of HDFS-8072

https://issues.apache.org/jira/browse/HDFS-8072

On Wed, Jan 10, 2018 at 2:20 PM, Xie Gang <xi...@gmail.com> wrote:

> Hi,
>
> Recently, we hit an issue that, there is a difference between the
> freeSpace of the datanode volume info and the OS df:
>
> For example:
> the jmx of the dn shows:
>
>  "VolumeInfo" : "{\"/....\":{\"freeSpace\":1445398864500,\"usedSpace\":228138206927,\"reservedSpace\":53687091200}}",
>
> But the df shows:
> /dev/sda 2146676656 <(214)%20667-6656> 253778008 1785508084 13% /...
>
> There is about 400GB gap which is regarded as Non DFS used. And the most
> strange thing is that, after I restart the dn process, the gap disappear.
> And after some days, the gap shows again.
>
> The yarn shared the same server of the dn and has some file cache. Could
> it be related?
>
> The direct cause is that the freeSpace from dn is quit different from the
> available space from df. After tracking down the code, freeSpace of the dn
> is from dirFile.getUsableSpace(). could it have some problem? Do we hit
> this issue before?
>
> Thanks,
> Gang
>
>
> --
> Xie Gang
>



-- 
Xie Gang