wrong remaining space reported by Data Nodes

[Ophir Etzion <op...@foursquare.com>]

Hi,

I have a cluster with about 200 Data Nodes.
when doing `hdfs dfsadmin -report` all nodes report less remaining space
then exists on the disks (as seen by 'df -h' on the data node machine).
it is less by about 2TB in every machine.
when restarting a DataNode it calculates the remaining space well again and
we get the 2TB back as remaining space.

any idea why this difference in the remaining space happens and how I can
solve it? (something other than doing a rolling restart on the nodes every
time this happens)

Thank you,
Ophir

Re: wrong remaining space reported by Data Nodes

[Ophir Etzion <op...@foursquare.com>]

Thank you very much for response!

Do you know if its just reporting badly in that command or will hdfs not
have that space available if it needs it?

Thank you,
Ophir

On Fri, Jul 22, 2016 at 9:35 AM, Vinayakumar B <vi...@huawei.com>
wrote:

> Hi
>
>
>
> You might be hitting https://issues.apache.org/jira/browse/HDFS-9530,.
>
>
>
> This will arrive soon in coming 2.7.3 Release. J
>
>
>
> -Vinay
>
>
>
> *From:* Ophir Etzion [mailto:ophir@foursquare.com]
> *Sent:* 22 July 2016 19:03
> *To:* user@hadoop.apache.org
> *Subject:* wrong remaining space reported by Data Nodes
>
>
>
> Hi,
>
> I have a cluster with about 200 Data Nodes.
> when doing `hdfs dfsadmin -report` all nodes report less remaining space
> then exists on the disks (as seen by 'df -h' on the data node machine).
> it is less by about 2TB in every machine.
> when restarting a DataNode it calculates the remaining space well again
> and we get the 2TB back as remaining space.
>
> any idea why this difference in the remaining space happens and how I can
> solve it? (something other than doing a rolling restart on the nodes every
> time this happens)
>
> Thank you,
> Ophir
>
>
>

RE: wrong remaining space reported by Data Nodes

[Vinayakumar B <vi...@huawei.com>]

Hi

You might be hitting https://issues.apache.org/jira/browse/HDFS-9530,.

This will arrive soon in coming 2.7.3 Release. ☺

-Vinay

From: Ophir Etzion [mailto:ophir@foursquare.com]
Sent: 22 July 2016 19:03
To: user@hadoop.apache.org
Subject: wrong remaining space reported by Data Nodes

Hi,

I have a cluster with about 200 Data Nodes.
when doing `hdfs dfsadmin -report` all nodes report less remaining space then exists on the disks (as seen by 'df -h' on the data node machine).
it is less by about 2TB in every machine.
when restarting a DataNode it calculates the remaining space well again and we get the 2TB back as remaining space.

any idea why this difference in the remaining space happens and how I can solve it? (something other than doing a rolling restart on the nodes every time this happens)

Thank you,
Ophir