You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by Xie Gang <xi...@gmail.com> on 2018/01/29 09:56:14 UTC

does it make sense to get remaining space by sum all the ones of the datanode

Hello,

We recently hit a issue that almost all the disk of the datanode got full
even we configured the du .reserved.

After tracking down the code, found that when we choose a target datanode
and check if it's good candidate for block allocation (isGoodTarget()), it
only checks if the total left space of all the volumes(the same type), not
each volume. This logic makes the reservation of each volume useless.
Is this a problem or do I have any misunderstanding?

final long remaining = node.getRemaining(storage.getStorageType());
if (requiredSize > remaining - scheduledSize) {
  logNodeIsNotChosen(storage, "the node does not have enough "
          + storage.getStorageType() + " space"
          + " (required=" + requiredSize
          + ", scheduled=" + scheduledSize
          + ", remaining=" + remaining + ")");
  stats.incrOverScheduled();
  return false;
}



-- 
Xie Gang

Re: does it make sense to get remaining space by sum all the ones of the datanode

Posted by Vinayakumar B <vi...@apache.org>.

in the latest code this is is not applicable. each storage is chosen and
space of that storage is checked.

On 29 Jan 2018 3:53 pm, "Xie Gang" <xi...@gmail.com> wrote:

> 2.4 and 2.6:
>
> public long getRemaining(StorageType t) {
>   long remaining = 0;
>   for(DatanodeStorageInfo s : getStorageInfos()) {
>     if (s.getStorageType() == t) {
>       remaining += s.getRemaining();
>     }
>   }
>   return remaining;
> }
>
>
> On Mon, Jan 29, 2018 at 6:12 PM, Vinayakumar B <vi...@apache.org>
> wrote:
>
>> in which version of Hadoop you are seeing this?
>>
>> -Vinay
>>
>> On 29 Jan 2018 3:26 pm, "Xie Gang" <xi...@gmail.com> wrote:
>>
>> Hello,
>>
>> We recently hit a issue that almost all the disk of the datanode got full
>> even we configured the du .reserved.
>>
>> After tracking down the code, found that when we choose a target datanode
>> and check if it's good candidate for block allocation (isGoodTarget()), it
>> only checks if the total left space of all the volumes(the same type), not
>> each volume. This logic makes the reservation of each volume useless.
>> Is this a problem or do I have any misunderstanding?
>>
>> final long remaining = node.getRemaining(storage.getStorageType());
>> if (requiredSize > remaining - scheduledSize) {
>>   logNodeIsNotChosen(storage, "the node does not have enough "
>>           + storage.getStorageType() + " space"
>>           + " (required=" + requiredSize
>>           + ", scheduled=" + scheduledSize
>>           + ", remaining=" + remaining + ")");
>>   stats.incrOverScheduled();
>>   return false;
>> }
>>
>>
>>
>> --
>> Xie Gang
>>
>
>
>
> --
> Xie Gang
>

Re: does it make sense to get remaining space by sum all the ones of the datanode

Posted by Xie Gang <xi...@gmail.com>.

2.4 and 2.6:

public long getRemaining(StorageType t) {
  long remaining = 0;
  for(DatanodeStorageInfo s : getStorageInfos()) {
    if (s.getStorageType() == t) {
      remaining += s.getRemaining();
    }
  }
  return remaining;
}


On Mon, Jan 29, 2018 at 6:12 PM, Vinayakumar B <vi...@apache.org>
wrote:

> in which version of Hadoop you are seeing this?
>
> -Vinay
>
> On 29 Jan 2018 3:26 pm, "Xie Gang" <xi...@gmail.com> wrote:
>
> Hello,
>
> We recently hit a issue that almost all the disk of the datanode got full
> even we configured the du .reserved.
>
> After tracking down the code, found that when we choose a target datanode
> and check if it's good candidate for block allocation (isGoodTarget()), it
> only checks if the total left space of all the volumes(the same type), not
> each volume. This logic makes the reservation of each volume useless.
> Is this a problem or do I have any misunderstanding?
>
> final long remaining = node.getRemaining(storage.getStorageType());
> if (requiredSize > remaining - scheduledSize) {
>   logNodeIsNotChosen(storage, "the node does not have enough "
>           + storage.getStorageType() + " space"
>           + " (required=" + requiredSize
>           + ", scheduled=" + scheduledSize
>           + ", remaining=" + remaining + ")");
>   stats.incrOverScheduled();
>   return false;
> }
>
>
>
> --
> Xie Gang
>



-- 
Xie Gang

Re: does it make sense to get remaining space by sum all the ones of the datanode

Posted by Vinayakumar B <vi...@apache.org>.

in which version of Hadoop you are seeing this?

-Vinay

On 29 Jan 2018 3:26 pm, "Xie Gang" <xi...@gmail.com> wrote:

Hello,

We recently hit a issue that almost all the disk of the datanode got full
even we configured the du .reserved.

After tracking down the code, found that when we choose a target datanode
and check if it's good candidate for block allocation (isGoodTarget()), it
only checks if the total left space of all the volumes(the same type), not
each volume. This logic makes the reservation of each volume useless.
Is this a problem or do I have any misunderstanding?

final long remaining = node.getRemaining(storage.getStorageType());
if (requiredSize > remaining - scheduledSize) {
  logNodeIsNotChosen(storage, "the node does not have enough "
          + storage.getStorageType() + " space"
          + " (required=" + requiredSize
          + ", scheduled=" + scheduledSize
          + ", remaining=" + remaining + ")");
  stats.incrOverScheduled();
  return false;
}



--
Xie Gang